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PREFACE 


“Currently in the period of dynamic indeterminism in science, there is 
hardly a serious piece of research, which, if treated realistically, does not 
involve operations on stochastic processes” (Neyman, J. Am. Stat. 
Ass. (1960) p. 625). It is thus natural to find a growing awareness and 
interest in this subject in scientific and technological studies everywhere. 

It is felt that, unlike statistical theory, a large variety of introduc- 
tory and intermediate level texts dedicated to stochastic processes are 
not available, and that there is need for more books at this level. This 
book which aims at the level between that of elementary probability 
texts and of the excellent advanced works on stochastic processes, has 
been written keeping in view students of diverse interests and back- 
ground. The book deals mainly with methods and applications. 

The prerequisites for using this book are a course’ on elementary pro- 
bability theory and statistics and a course on advanced calculus. Some 
of the basic mathematical concepts and techniques used (in the sequel) 
have been treated in some details in Chapter 1. The theoretical results 
developed have been followed by a large number of illustrative examples. 
These have been supplemented by a large number of exercises, answers 
to most of which are also given. Lists of references are provided for 
the benefit of those who need more details or wish to pursue the subject 
further. 

It isexpected that the book would be suitable as a text book for advanced 
` undergraduate, post graduate and research level courses in Statistics, 
Applied Mathematics, Operations Research, Computer Sciencé, Manage- 
ment Science etc. as also for graduate and beginning research level 
courses of study in Physics, Life Sciences and different branches of 
Engineering. The course could be covered in one ог two semesters 
depending on the interest of the users and on the extent or coverage 
desired. Researchers who need knowledge of applied probability and 
applied stochastic processes would also profit from using the book. 
The illustrative examples spread over areas, such as, biological sciences 
to management, sciences would lend a special appeal to not only those 
who wish to pürsue the subject but also to those who wish to go 
for applications. Some materials, specially on applications from some 
of the most recent papers are also, included. There isa somewhat 
exhaustive chapter on stochastic processes in Queues, and this portion 
could be used also for a course on queueing theory. The material 
presented in the book would, it ishoped, be suitable, in general, for 
Students having some mathematical maturity who wish to gain'an 
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understanding of the subject and to see how the theory works. f 

The book has grown out of the author’s long 30 years’ experience of 
teaching and research. The suggestion of writing a book on the lines 
of his lecture notes, was first made to him by groups of ‘students who 
took such a course given by him at the University of Montreal, Canada. 

I look forward to receive omments and suggestions on the werk from 
students, te.chers and researchers. 

{remain intellectually indebted to all those workers whose works have 
stimulated my interest in the subject. In the present venture, encoura- 
gements and suggestions have been received from a number of learned , 
scholars both in this country and abroad. Iam thankful to them all. 

I am indeed most grateful to Professor N.U. Prabhu, School of 
Operations Research, Cornell University, U.S.A. and to Professor R. 
Fortet, Université de Paris, France, both of whom kindly read the 
manuscript thoroughly and made very valuable comments and offered 
constructive suggestions. I recall also the appreciative interest shown 
and the constant encouragement given by Professor Prabhu. 

Iam indebted to Professor M.B. Priestley, University of Manchester 
Institute of Science & Technology, U.K. and to Professor R.M. Loynes, 
University of Sheffield, U.K. for having glanced through the manuscript 
within a short time and for making useful comments, specially with 
regard to coverage. 

Y received considerable assistance with proofreading from some of my 
former research students and present colleagues, namely, Dr S.B. Nandi, 
Dr. A. Borthakur, Dr. H.K: Baruah and Mr. D.C. Jain as well as from 
my elder son, Deepankar; and help in many other ways from my 
younger son, Shubhankar and two daughters, Shakuntala and Alaka- 


nandaa. Acknowledgement is also made of the excellent typing by S. 
Borah and A. Choudhury. 


JYOTIPRASAD MEDHI 
Gauhati, India З 
August 1981 
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CHAPTER 1 


INTRODUCTION 


1.1 Generating Functions 


A. Introduction 

In dealing with integral-valued random variables, it is often of great 
convenience to apply the powerful method of generating functions 
(Feller, Vol. 1, Chapter XI). Many stochastic processes that we come 
across involve integral-valued random variables, and quite often we 
could use generating function in their studies. The principal advantage 
of its use is that a single function (generating function) may be used to 
represent a whole set of individual items. 


Definition. Let ay, а;, а,, ... be a sequence of real numbers. Using a 
variable s, we may define a function 
оо 
А (5) = а, + ays + as? +... = X apsk. (1.1) 
Кеб 


If this power series converges in some interval — sy < s < ss, then A (s) 
is called the generating function of the sequence dg, a}, à» .... The 
variable s itself has no particular significance. Here we assume s to be 
real but a generating function with a complex variable is also sometimes 
used. 
Differentiating (1.1) k times, putting 5 = 0 and dividing by k!, we 
get ак, i.e. 
1 Гад 


SILEF |, (1 2) 


ак 


B. Probability Generating Function : Mean and Variance 
Suppose that Y is a random variable which assumes non-negative 
integral values 0, 1, 2, . .., and that 


Pr (X =k} — py, k —0, 1,2, ....Хрк= 1. (1.3) 


If we take a; to be the probability pj, k = 0, 1, 2, ... then the corres- 
ponding generating function P(s) of the sequence of probabilities {рк} is 
known as the probability generating function (p.g.f.) of the random 
variable №. It is sometimes also called the s-transform (or geometric 
transform) of XY. 
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We have 


Р(з)— X ps* Ga 
| K-0 


— $ Pr(X— K) s*— Уб; (1.42) 
к=0 
where (5х) is the expectation of the function s* (a random variable) of 
the random variable X. 

The series Р(х) converges for atleast —] < s <1; clearly P(1) -- 1. The 
first two derivatives of P(s) are given by 


РЧ) = У kp, — 248 (1.5a) 
kal 
and 
Р") = X k(k — 1) ризк. (1.5b) 
kel 


The expectation A(X) is given by 


E(X)- d kp, — P'1). (7.6) 
T! 
"We have 
E(X(X — 1) — z k(k — 1) pk = P''(1) (1.7) 
and 
г RX?) = E(X(X — 1)}+ E(X) = Р''(1) + Р'(1). 
Hence 


var(X) АХ) — [AXE 
Р (Me P'CDy ру). (1.8) 
If, as s — 1, X kp, diverges then we say that F(X)— P'(1) — o and if 
®К(К — 1) py diverges then P"(1) — oo and var(Y)- = ©. 
Equation (1.6) gives the mean and (1.8). the variance of Y in terms 
of the p.g.f. of V. In fact, moments, cumulants etc. can be expressed in 


terms of generating functions. More generally, the Ath factorial moment 
of Y is given by 


SE UAM > dk 

EX -1)...(Х- KADY = SG РО for k 1, 2,.... 
Note that (e!) is the moment generating function and P(Y 1) is the 
factorial moment generating function. Further, the p.g.f. P(s) = E(s*) 
is a special case of the characteristic function £(e/*) or the Fourier 
transform of V. 


1 
Note also that | P(s) ds Ez ) 


° 
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Example 1(а). The sequence fa, = (1/k!)) has the generating function 


A(s) 5 k = 58 
$) "= US РО 7 а 
( k=0 ко Е! 


Here (ay) is a sequence of real numbers. This does not represent probabi- 
lities of any random variable. 


Example 1(b). Poisson distribution: Let X be a Poisson variate with 
probability function 


€ k 
peer ed] = сек m А Ж 


The p.g.f. of Poisson distribution (or of the random variable X) is 
given by 


-— k 
iets uited SALOUAM 
ke0 


k! 


y 


> 


0 
— exp (—A) exp (As) 
= exp (s — Dy. 
Since P'(s) = Л exp (A (s — D) and P"(s) =: A? exp (s — Dj, we “have, 
from (1.6) and (1.8) 
F(X) = А and var (Y) --А. 


Example l(c). Geometric distribution: Let X be a random variable 
with geometric distribution {pi}, where 
Pes РЫХ) = ар) ^ ke0,1,2;... 


pt+q=l. 
The p.g.f. of X is 
Р) Ў pis* — р $ (qs) — S 
k=0 k=0 1=95 
We have 
, Pq „ 2pq? 
ae. реу d 
P) (1 — qs’ @ (1 -- gs)? 


From (1.6) and (1 .8) we have 


йу Ex PIU mo P e ea 
E(X) Р'(1) 1-4 р 
апа 2g? 2 
=P" P'(1) — [P'(1 20 1 (2y- £5. 
var (X) = Р”(1) + P'(1) — [P'(0) p Up Xp 


An example of a random variable Y which follows geometric distribu- 
tion is given below. Consider a sequence of Bernoulli trials (i.e. indepen- 
dent trials with only two possible outcomes “success” and “failure at 
each trial, the probability of success being p and that of failure being 
q--1-— р). Then the number X of failures preceding the first success 
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has geometric distribution with Pr (X = k}=g*p(k = 0, 1.2,.. de X is 
also called the waiting time for the first success. Note that the continuous 
analogue of the geometric distribution is the exponential distribution. 


Example i(d). Binomial distribution: Let X be a random variable 
with binomial distribution (number of successes in п Bernoulli trials) 
with 


pic Pr {xX =k} = ig )» get o. s 2. ud 


RVG =, 
The p.g.f. of Y is 


LJ n 
Р(х) = X p,s^ 2 f ) pkqr-k sk = (4 + sp)". 
к= k=0 

Thus 

FUN) P'(1) -np 

and 


хаг (V) Р) + P'(D- [P' qn 
п(п-— Dp? © np = np? 
npq. 
Example Це). Let X be a random variable with p.g.f. P(s). To find 


the p.g.f. of the random variable Y = mX + n, 
and nm z^ 0. 


Let 


where m,n are integers 


Py (s) and Py (s) 
^e the generating functions of Х and Y respectively. We have 
Py (8)  E(s'] = Етну 
= OE (smt sm — sn. FE ((smyxi 


= s" Py (s), 


From the above examples, we sce how a single generating function 
P(s) may be used to represent a whole set of probabilities 


Py Pri XSTACK 0032: 


In these examples we were concerned with the problem of finding P(s) 
for a given set of pgs. 

In many cases the reverse problem arises: to determine px from a given 
pel Pis). Many situations arise, where it is easier to find the p.g.f. 
Р(х) of a variable rather than the probability distribution {p,} of the 
variable. One proceeds to find first the p.g.f. P(s) and then to find the 
probabilities p; from the function Pis). Even without finding the pes 
one can find the moments of the distribution from Р(х). 
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Determination of py from P(s): py can be (uniquely) determined from 


P(s) as follows: 
px can be found from F by applying (1.2), i.e. 


d* д) i 
ES di: Jon! 

pk is also given by the coefficient of s* in the expansion of P(s) asa 
power series in s. 

When P(s) is of the form P(s) — U(sY/V(s), it may be convenient to 
expand Ps) in a power series in 5 first by decomposing P(s) into partial 
fractions. Suppose that s,,...,5, are the distinct roots of V(s), i.e 
V(s) = (s— $)...'s— Sr) (apart from a constant factor c which, for 
simplicity, we take to be equal to 1), then P(s) can be decomposed 
into partial fractions as 

dr 
+ i 


pss ! "— 
i S 5 Se 


where a’s can be determined. It may be verified that 


a; = — U(si V' (si). 
Now 


Hence px cocflicient of s* in P (s) 


ау а ar 
j* Б мет 5° v» Ез: 
Kl 5н sett 


C. Sum of a fixed number of Random Variables 
Let Y and Y be two independent non- negative integral-valued random 
variables with probability distributions given by 
Pr(X = К} = aj Pr(Y =j} = bj. 


X i Yis a random variable. The event {Z = ғ) сап 


The sum Z — 
lly exclusive ways with corresponding 


happen in the following mutua 


probabilities: 
(X =0 and Y =r) with probability ар, 


с ander 1) with рту abr- 
w =t and Y--r -» nd probability ањ. "n 


(х = r and Y == 0) with probability arbo- 


; Hence the distribution of Z is given by 
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ĉr = Pr {Z =r} = a, + aibri + -.. + ab, = A aibi (1.9) 


This new sequence (cj which results from a combination of the two 
sequences (aj, {bj} in this particular way indicated by (1.9) is called the 
convolution of {аку and (bj) and is denoted by 

{сг} = lansibi. 


Let A(s), B(s), C(s) be the p.g.f. of Y, Y and Z respectively. 
Then from (1.9) it follows that 


C(s) = A(s)- B(s). (1.9а) 
This also follows from the fact that 
C(s) = E(sX*Y) = E(s*) E(sY), 
= A(s)- B(s), 


because of independence of Y, Y. 
We have thus the following theorem: 


Theorem 1.1. The p.g.f. of the sum of two independent random variables 
X and Y is the product of the p.g.f. of X and that of Y. 

The result will also hold good in the case of the sum S, of n 
non-negative independent integral-valued random variables Жуз OT 
ie. the p.g.f. of S, is the product of the p.g.f.'s of Key Kas Meng 

In particular, we have the following: 


Theorem 1.2. The sum Sn=Xi+.. +X, of a fixed number n of 
identically and independently distributed (i.i.d.) random variables X; has 
the p.g.f. {P(s)}", each X; having p.g.f. P(s). 

The results stated above are of considerable importance. These can be 


employed to find the distribution of the sum of a number of independent 
random variables. 


Example 1(f). Sum of independent Poisson variates: Let X. p X be 
two independent Poisson variates with parameters (means) Ai, A, respec- 
tively. To find the distribution of the sum Z = X, + X, 
The p.g.f. of X is exp {A;(s — 1)}, i= 1,2. 
Hence the p.g.f. of Z is 
[exp {Ai (s — DJ] [exp {Ag (s — 1)}] 
= exp (Qs + Aj)(s — 1). 

But this is the p.g.f. ofa Poisson variate with parameter A, + A,. Hence 
the sum of two independent Poisson variates is a Poisson variate. More 
generally, if X;,i=1,2,...,n aren independent Poisson variates with 
parameters À;, then the sum S = Xy + X, --...-- X, is a Poisson variate 


п 
with parameter X A; 
i=l 
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Example 1(8). Let X;, Х be two independent random variables each with 
the same geometric distribution, Pr (X; = Aj —q*p, i — 1, 2. 

The p.g.f. of Y; is р/(1 — qs) for i = 1,2. Thus, the sum Z = X, + X, 
has the p.g.f. 


epos [il ore 
Fo - (ia) oS ( r Jes 


Pr {Z = r} = coefficient of s in P(s) 


Hence, 


ae | A E 
= (d a јар =е + igp, r= 0, 1,2, .... 


This gives the distribution of the sum 2. If Y, denotes the number of 
failures preceding the first success and Y, the number of failures follow- 
ing the first success and preceding the second success, then Z = Хх, + X, 
denotes the number of failures preceding the second success. 

The sum S, = +... БА of r i.i.d. geometric variates denot-s the 
number of failures preceding the rth success. It has the р.в. 


P(s) = (+ 2y It follows that 


Pr (S, = К} = coefficient of s* in P(s) 


(161) э, 'k =0, 1,2,... (1.10) 
r+k-l1 
> rgk = 
and (( k ) ora 1 (1. 10a) 


Here r is a positive integer. But the quantity given by the r.h.s. of (1.10) 
is non-negative and (1. 10a) holds for any positive г. The distribution 
(1.10) for r > 0 is known as negative binomial distribution with index r, 
and mean rg/p. In particular, when r is a positive integer, it is known 
as Pascal distribution. 


Example 1(h). If X, Y are independent r.v.'s. with generating functions 
A(s), B(s) respectively, then the generating function D(s) of -X-—Yis 
given by 
Dis) = E (5X) = E (s* (57!) = E {5%} E {0/9} 
because of independence of X, Y; thus 
D(s);= A(s) В(1/5). 


D. Sum of a Random Number of Discrete Random Variables 

In Section C we considered the sum Sn = X, +--+ X, of a fixed number 
of mutually independent random variables. Sometimes we come across 
situations when we have to consider the sum of a random number N of 
random variables. -For example, if X; denotes the number of persons 
involved in the ith accident (in a day їп а certain city) and if the number 
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N of accidents happening on a day is a random variable, then the sum 
Sy = X, ...-- Ху denotes the total number of persons involved in 
accidents on a day. 

We have the following theorem for the sum Sy. 


Theorem 1.3. Let X;,/=1,2,..., be identically and independently 
distributed (1.1.4.) random variables with Pr (X; = k} -= p, and p.g.f. 
P(s) == pys* fori —1,2,... (1.11) 
k 


Let Sy =X, +...+ Ху, (1 12) 


where N is a random variable independent of the Хз. Let the distribu- 
tion cf N be given by Pr {N == n) — g, and the p.g.f. of N be 


G(s) -= Xg,s^. (1.13) 
Then the p.g.f. H(s) of Sy is given by the compound function G(P(s)), 


i.e. 


H(s) m Pr(Sy =j} si = G(P(s)). (1.14) 


Proof: Since N is a random variable which can assume values 0,1253: 12 
the event Sy — j can happen in the following mutually exclusive ways: 


тапа ole Se Nn =j forn=1,2,3,.... To meet the situa- 
tion Л = 0, let us define Xa = So =: 0 so that 
Xy Ху tat X= Sy п> 0. 


We have hi = Pr {Sy =j} = 5 Pr(N —n and S, =) 
n=O 


Since N is independent of Хгѕ and therefore of 5 


n 


hy = X Pr(N = n} Pr(S, = j} (1.15) 
neu) 

= X gn Pr(S,— j}. (1.16) 
ne 


(The inclusion of the value 0 for n can be justified since Pr(S, = 0) = 1 
and Pr{S, > 0) = 0.) 
The p.g.f. of the sum Sy =X,+...+ Xy is thus given by 
H(s) = Х hjsi =È Pr {Sy =j} s (1.17) 
j ere 
= [E gn Pr(S, =j} s/ (1.18) 
j=0 n=O 


=Z [E Pr {S, =j} s] gn 
" J 
= в, [PI = GG), 


since the expression [P(s)]" is the p.g.f. of the sum S, of a fixed number 
n of i.i.d. random variables (Theorem 1.2). A 
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Corollary: The mean 


Еур ЕХ EQN}. (1.19) 
We have E(Sy) = Н'(з) [pny = P'()G (P(5)]..., 
= Р'(1) G'(P(1)) = P'(1) G'(1) 
= БХ) EQN}. 


Note. If P(s) and G(s) are two p.g f.’s, then G(P(s)) is also a p.g.f. 


Example 1(i). Compound Poisson distribution: Yf N hasa Poisson distri- 
bution with mean А, then G(s) —exp(A(s— 1)) and hence the sum 
X,+...+ Ху has p.g.f. 

H(s) = G(P(s)) = exp (^ [P(s) — 1])- (1.20) 
The distribution having a generating function of the form exp(A[P(s) — 1]}, 
where P(s) is itself a generating function is called compound Poisson 
distribution. 

The mean is given by 
E{Sy} = Е{Х EGN) = ^ Е(Х;). 


E. Generating Function of Bivariate Distribution 
Bivariate Distribution: Suppose that Y, Y is a pair of integral-valued 
random variables with joint probability distribution given by 


Pr{X=j, Y= =рьј, К =0,1,2,.52 рь=1. (1.21) 
Ak 


The marginal distributions are given by 


Pr{(¥=j}= 3pa—f, j-012.. (1.22) 


Pr{(Y=k}= раев К=0,1,2,.... (1.23): 
=0 


If (j, k) is any point at which f; 40 (і.е. f; > 0), then the conditional 
distribution of Y given XY = j is given by 


рухе Ts ex. Ра, fies. o. 
(1.24) 


Conditional Expectation 
The conditional expectation E(Y | X = j} is given by 
BIY | Xj ӘР СЕ | X 2j) 


Bikes $501.25... 1.25) 
-IK7 T j (1.25) 
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For X given but unspecified, we note that E(Y | X) isa random variable 
У Крук Р , 
which assumes the value EX with Pr(X =j} = /; > 0. 
J 
Hence the expectation of the random variable E{Y | X} is 


EĻEĻY | X] =X E(Y | X =j} PHX =j} 


= У Укр 
k 
Mill ee Pr(Y =k} = E(Y). (1.26) 
In the same way we can prove that 
E[E(Y? | Xj] = Е[Ү?]; (1.27) 
more generally, for any function $(Y) whose expectation exists, 
ELE(&CY) | XJ] = EON. (1.28) 


Note: The results (1.26-1.28) which are given here for discrete random 
variables Y, Y will also hold, mutatis mutandis, for continuous random 
variables. 


Bivariate Probability Generating Function 


Definition. The probability generating function (bivariate p.g.f.) of a 
pair of random variables X, Y with joint distribution given by (1.21) is 
defined by 


Pons) E Piksi Ss, (1.29) 


Sı, 5 being dummy positive real variables chosen so as to make the 
double series convergent. We have the following result. 


Theorem 1.4. (a) The p.g.f. A(s) of the marginal distribution 
Pr (X = j} is given by A(s) = P(s, 1). 
(b) The p.g.f. B(s) of Pr(Y — K) is given by 
B(s) = Р(1, s). 
(c) The p.g.f. of (X 4- Y) is given by P(s, s). 
Proof: Since the convergent double series (1.29) consists of positive 


real terms, the change of order of summation is justified. 
(а) We have from (1.29) 


P(s1)— X разу = X X J 
2,8 е p EX Paks 
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= B { E ра} 5! £X Pr{X¥=/}s/, from (1.22) 
j=0 k=O j-0 


= A(s), which is the p.g.f. of Y. 


(b) can be proved in the same way. 


(c) We have 
> | = m 
Р(5, 5) = = рњ5/# = 5 E Primas” 
j, ken m-0 r=0 


as can be casily verified. 


: m m 
Now PriXY--Y-—m)- X РХ =r, Y =m— у = 


r=0 r 


У Pr mars 
-0 
Hence 


P(S, 5) = 5 [Pr (X. i- Y= ту] 5", 


m=0 


which shows that P(s, 5) is the p.g.f. of YoY. А 


Remarks: (1) If X, Y are independent, 
Dik — Pr(X =j, У = К} 2 Pr {X --j; PriY — k} 
and then P(s,, Sa) = A(Sy) В(5) = Pisi 1) PUL, sa) 


and conversely, from this relation follows the independence of Y and Y. 
(2) The probabilities pj, can be (uniquely) determined from Pisi, s;) 


as follows: 
1 pak Pus, s,) 
"m ul mp а, 


Example 1(j). Consider a series of Bernoulli trials with probability of 
success p. Suppose that X denotes the number of failures preceding the 
first success and Y the number of l'ailurcs following the first success and 
preceding the second success. 
The joint distribution of №, Y is given by 
Bj —Pr(Y-:j, Y-- К) —qii^p,  L,&k—0,1, 2,..: 
the bivariate generating function is 


= 
P(s,5)- 2 Pye sisg— 2 qg ps 5% 
j,h=0 ik 


=p E (05) CE (595 


jon] - 
р? 
(=й 80 
The p.g.f. of (Y; is given by 
Р - р? TN: 
A(s) = Р( 1) (1—sq)(1—4) 1—95" 


\ 
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The p.g.f. of X -- Y is given by 


P (зз) =( Ax) 


1.2 Laplace Transforms 


A. Introduction 

Laplace transform is a generalisation of generating function. Laplace 
transforms serve as a very powerful tool in many situations. They pro- 
vide an effective means for the solution of many problems arising in our 
study. For example, for the solution of difference equations the trans- 
forms provide a very effective method. In the study of some probability 
distributions, the method could be used with great advantage, for it 
happens quite often that it is easier to find the Laplace transform of a 
probability distribution rather than the distribution itself. Here we 
shall mention some of the important properties of the Laplace trans- 
forms and illustrate the method by means of some examples (see 
Widder). 


Definition. Let /(7) be a function of a positive real variable t. Then the 
Laplace transform (L.T.) of f(t) is defined by 
1 


F(s) = ] exp (—st) f(t) dt (2.1) 


9 


for the range of value of s for which the integral exists. 
We shall write F(s) = L{f(t)} to denote the Laplace transform of /(t). 


Example 2(a). 
@ Xi Fis) = [exp (— st) dt = Ms (s > 0) 
0 
If Д) =, Ks) = [ texp(—st)dt=1/s? (52 0) 
E f(t) — i"(n  — 1), F(s) = |" exp ( — st) dt = Г(п + 1)/s"*! 
о 


(5 > 0) 


Gi) Let f(t)=e, then F(s) = | exp ( — st) exp (at) dt 


0 


| 


1/(s — a) (з > a) 
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(ii) Let f(t) = sin í, then F(s)= | exp ( — st) sin t dt 
> 0 


EEI (s >0) 


(iv) Let Д0) = e~ 1, then F(s)= [ exp ( — st) exp ( — t) t" dt 


=f exp (— t(s + 1)) t^ dt 
0 


Г(п+ 1) 


= рту (n > —1). 


B. Some Important Properties of Laplace Transforms 
We state the results without proof; the reader may refer to any standard 


work for proof. 
1. . Linearity property: 1 c's are constants, айа 
Lifi (0) = Fils), $= 1, 2, ... m 
then Lie, ft) +... сад} = e FG) H- E сЁ), - (2.2) 
3 L(21 — 5) = 2/s? — S/s. 


e.g. 
2. First translation property: 
L((exp (—at)) (0) = F(s а), a>, Q;3)« 
eg. L(exp (— 00 = 5 E 


(2.3) holds also for such negative values of a for which thecorresponding 
integral exists. 


3. Second translation property: 


ж 60-— p ee onn 20 
then L(G(1)) = exp ( — as) F(s). (2.4) 


4. Change of scale property: 


L(f(at)) = (1/a) F(s[a). (2.5) 
Since Tsin ne wer we have 
a 


х 1 
L(sin at} = (1/4) Gai sul ur 
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Example 2(b). 
L((exp ( — at)) (at)"} = L{fat)} = (1/a) F(s[a) 


Inl) _ a T(n4-1) 
a((s/a) + 1) — (s--aym ' 


5. L.T. of derivatives: L{f'(t)} = sF(s) — f(0). 


Example 2(c). If f(r)— sin t, then F(s) = 1/(s? 4- 1) 
14соз t} = LCf'(0) = s/(s* + 1). 


6. L.T. of integrals : [ә ax} = F(s)/s. 


If S(t) = cos t, then F(s) = s/(s* + 1). 


Since sint =| cost dt, L (sin t) = 1/(s* + 1). 
0 


7. Limit property: Initial value theorem 
lim f(r) —lim sF(s), 
ie s—>0 
whenever the limits exist. 
Example 2(4). Let f(r) = exp(— 1), then F(s) = 1/(х + 1). 
We have lim f(t) — 0, asis lim sF(s). 
[жул so 
8. Limit property: Final value theorem 
lim f(t)= lim sF(s), 
t—>0 s—>a 
whenever the limits exist, 
Example 2(e). Let f(t) — exp(—t), then F(s) — 1/(s-+ 1) 


lim f(t) 1 and lim sF(s)— lim — — ] 
1—0 ` sn 505-1 1 


(2.6) 


(2.7) 


(2.8) 


(2.9) 


The properties (7) and (8) enable one to find the behaviour of f(r) as 
t > оо (or as г > 0) from the behaviour of Дз) as s > O(or as s > oo). 
There are many other such results of Breat generality; they are covered 


under the name of Tauberian theorems. 


C. Inverse Laplace Transform 


Definition. If F(s) is the L.T. of (1), Le. Lf} = F(s), then f(t) is 
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called the inverse Laplace transform of F(s). For example, the inverse 
Laplace transform of F(s) = 1/s* is f(t) = t. There is an inversion for- 
mula which gives f(t) in terms of F(s) and given F(s), the formula could 
be employed to find f(t) (uniquely). 

Extensive tables of Laplace transforms are also available and reference 
may be made to them, whenever necessary, to obtain either the L.T. of a 
function f(t) or the inverse L.T. of a function F(s). 


1.3 Laplace (Stieltjes) Transform of a Probability 
Distribution or of a Random Variable 


Let X be an non-negative random variable with distribution function 
F(x) = Pr (X € x). 


The Laplace (Laplace-Stieltjes) transform f*(s) of this distribution is 
defined, for s > 0, by 
уч) = | exp(— so aeo (3.1) 
0 

(assuming that the range of X is from 0 to оо). 

We shall say that (3.1) also gives the ‘Laplace (Laplace-Stieltjes) 
transform of the random variable X". 
We have 


f*(s) = E(exp( — sX)} (3.1a) 
and f*(0) = 1. (3.2) 


Suppose that X is a continuous variate having density f(x) — F'(x). 
Then from (3.1), 
= 
учо) = [exo (— з) fax (3.3) 
0 
(this is the “ordinary” Laplace transform L(f(x)) of the density function 
f(x). Incase X is an integral-valued random variable with distribution 
рк = Pr(X = k}, k=0, 1, 2,..., and p.g.f. Р(5) = > prs, we can stretch 
the language and define the L.T. of X by 
f*(3 = E {exp (—sX)} = Р{ехр (— 5)). (3.4) 


Thus in case of a discrete random variable assuming integral values 
0, 1, 2,3,..., the L.T. of the variable differs from its p.g.f. only by a 
change of variable: exp(—s) in the former replaces s in the latter. Thus 
there is a close analogy between the properties af Laplace transforms and 
those of generating functions or s-transforms. 
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The Laplace Transform of the Distribution Function in 
Terms of that of the Density Function 

Let X be a continuous (and non-negative) r.v. having density function 
f(x) and distribution function 


Pr{X < x} = F(x) = J f(t) dt 


The (ordinary) Laplace transform of the distribution function F(x) is 
ЦЕХ) = | exp( — sx)F(x)dx 
0 П 


and the Laplace transform (Laplace-Stieltjes transform) of the distri- 
bution (or of the r.v. X) is 


учо) = | exp( — здана) = f exp C — sx) одах = ЦУ). 
0 ° 
We have from (2.7) 


е x 


L{F(x)} = [ec = sf | soda} esi { J лода} 


о 0 


= f*(s)/s. (3.5) 
The relation can also be obtained by integrating by parts the relation 


(3.1). 


A. Mean and Variance in Terms of (Derivatives of) L.T. 
We note here that differentiation under the integral sign is valid for 
the L.T. given by (3.1). 


Differentiating (3.1) with respect to s, we get 


ie) = [= exp( = зда) 


o 


= — [ xekp(-— sx)dF(x), 
9 


d2 T А 
РТА *(s)=(-1 Я! x? exp( — sx)dF(x) 
o 
and, in general, 


7 0) =( zu x” exp( — sx)dF(x). (3.6) 


0 
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We can use the above relation to find E(X”), the nth moment of X; we 
have 


(—1y [ero] = | {exp (— sx)} x" dF(x) for s =0 


= | x" dF(x) = Е(Х"). 
0 
In particular, 


ET =з [ero]. | (3.7) 
E(X?) = [= f*(s) 197 (3.8) 
qud var(X) = arol, [{- „= rol pr. (3.9) 


Example 3(a). A special kind of discrete distribution: 
Suppose that X is a random variable whose whole mass is concentrated _ 
in one single point, say, point a. This implies that the variable is ‘almost 
always’ equal to a. i.e. Pr(X = a) = 1 and Pr(X + a) = 0. Its distribution 
function is given by 


F(x) = РХ x) = 0 fo x<a 
EI for x >a. 
We have E(X) = a and var(X) = E(X?) — (F(X))? = а? — а? = 0. 
The L.T. of the distribution is given by 
f*(s) = E(exp( — 5X)) = exp(— sa)Pr(X = а) = exp(— sa). (3.10) 


This is the L.T. of Dirac function located at a. 


Example 3(b). Poisson distribution: Let X have Poisson distribution 
with mean A. Its p.g.f. is P(s) = exp{A(s — 1)}. 
From (3.4), its L.T. is given by 

f*(s) = Plexp(— s)) = exp[Mexp( — s) — 1}. (3.11) 
Example 3(c). Negative exponential distribution: Let X have the negative 
exponential distribution with parameter A. Its density function is 


fix) = А exp( — Ах), ASO 0<х<о, (3.12) 
=0 3 x«0, 


2(45-95/1982) 


18 STOCHASTIC PROCESSES 


2 4 6 
х 


Fig. 1.1. The negative exponential density function 
for some values of à. 


The L.T. f*(s) of the r.v, X is 
/*(5) = [хк — sx)) {A exp( — Ах)} dx = A/(s + А). 


We have 
СП з À di ел. VON 
qu (5) = — GEN aes (s)= EFH 
Hence from (3.7) and (3.8) 
A 1 24 0:2 
вю -[- 5] s-5-5 6-19 


2 
and var(X) = 53 — 33 — в (3.14) 
The distribution is also known as simply exponential distribution. 


Example 3(d). Uniform (rectangular) distribution: Let X have uniform 
distribution in (0, 1). The density function of X is 
Дх) =1, 0<х<1 
= 0, otherwise. 
The L.T. f*(s) of the r.v. X is 
2 il 
f*(s) = [ex — sx) f(x) dx = | exp( — sx) dx = (1 — exp( — s)}/s. 
о 


0 
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The mean of X is à and the variance is 1/12. 


Example 3(e). Gamma distribution: Let X have a two parameter gamma 
distribution with parameters А, k (А > 0 is the scale parameter and k > 0 


is the s/tape parameter). 
The density function of the r.v. Y is 


kyk-1 ae 
f(x) = AXES exp Ax S A) x0 


=0, x<0. (3.15) 
The L.T. /*(s) is given by 


LI 


[Xs f exp( — sx) f, к (x) dx 


v 


= (AK/D(k)) f [exp{ — (s + Ах} х*^—1 dx 


0 


= буга) [SPOR ar, (putting (у +A х0) 
0 


(s +A) 
s C 
=(; з) (3.16) 
We have 

d kak 

ag Г) = — +e 

d _ kk +1) ae 
p /*б) = WS AEN 


Hence from (3.7) and (3.8) 


| Е ) 
rw- -[- 3] E 
s LACE T)AE Ke T) 
BQ uei 
and 
^ 2 
var (x) = EE 1) - (5) = (3.18) 


When A -- 1, we get single parameter gamma variate with density 


xk-lexpi( —x) Д 
——т(® (x > 0): 


and its Laplace transform is 1/(s + 1)*. 
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When k = 1, the gamma distribution becomes the negative exponential 
distribution with density Aexp(— Ax) and having Laplace transform 
Afs +A). Another special case is A = 1/2 and k = n/2, where n is a positive 
integer. : The density then becomes (for x > 0) ; 
x"i?-1 exp (— x/2) 
2"? T(n/2) 

and the random variable is then said to have a chi-square (x?) distribu- 
tion with п degrees of freedom. The L.T. of the density is (1 + 25)-"/. 

Figure 1.2 gives the density function of gamma distributions for some 
values of A and k. 


Fig. 1.2. The gamma density fy x(x) for some values of A, k. 


The gamma distribution has considerable scope for practical 


application; many empirical distributions can be represented roughly by 
a suitable choice of the parameters A, k. 


Example 3(f). Erlang distribution: Writing Ak for A, we have a gamma 
distribution with parameters Ak, k having density function 


ЛК) k-1 =X ы 
fi, " GERIT вар С— 0), 0 (3.19) 


The variable having this for its density function is said to have Erlang 
(or Erlang—k) distribution and is denoted by Ey. In fact, when k is a 
positive integer, gamma distribution with density к, is known as Erlang 
distribution. 

The L.T. of Erlang distribution is 


АК NE. 
(; + x) : (3.20) 
further, E(E,) = КАК = 1/0; var (Ex) = k/(Ak)? = VEN. (3.21) 
B. Three Important Theorems 


We state without proof three theorems concerning L.T. of probability 
distributions. 


BAA 
i 


eN 
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Theorem 1.5. Uniqueness Theorem: Distinct probability distributions 
have distinct Laplace transforms. 

It has the important implication that a probability distribution is 
recognizable by its transform. If the L.T. of a random variable X is 
known, then by identifying this with the form of L.T. of a distribution, 
one can conclude that Y has the corresponding distribution; if this form 
does not resemble with the form of a L. T. of any standard distribution 
one can proceed to find the inverse of the L.T. (some numerical methods 
have also been developed) to get the distribution. Even without 
finding the inverse, i.e. the form of the distribution, one can compute the 
moments of the distribution. But for this theorem, perhaps, L.T. would 
not have been able to play the role it now plays in the study of probabi- 
lity distributions. 


Theorem 1.6. Continuity Theorem: Let {Х„}, n = 1, 2,...be a sequence 
of random variables with distribution function {F,} and L.T. {f,*(s}). If 
as n — co, F, tends to a distribution function F having transform /*(s), 
then as n-> co, f,*(s) — fis) for s > 0, and conversely. 

This theorem can be used to obtain the limit distribution of a sequence 
of random variables. 


Theorem 1.7. Convolution theorem: The Laplace transform of the con- 
volution of two independent random variables Y, Y is the product of 
their transforms. 

The integral 


| g(x — у) f(y) dy 


(denoted by f*g) is called the convolution of the two functions f and g. 
The convolution U of F and G is given by 


UG) = [ 66 — » aro» (3.22) 
and L{U(x)} = LLG} HEO. (3.222) 


In case of discrete random variables the result is essentially the same as 
that stated in (1.10) for generating function of the convolution of two 
random variables. In case of continuous random variables we have now 


the analogous result. | 
The above result is equivalent to the assertion that if Y, Y are two 


independent random variables, then 
E(e- 9*1 = Е(ехр (— sX) exp (— sY)} 
= E(exp (— 520) Е{ехр (— sY))- (3.23) 


) The result can also be stated as: 


4 
C.E.R.T., West Benga 
в. LA ДЕЙ == & 
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Theorem 1.8. The L.T. of the sum of two independent variables is the 
product of the L.T.’s of the variables. 

We can use the result to find the distribution of the sum of two or 
more independent random variables, when the variables are continuous 
or discrete. The method of generating functions is applicable only when 
the variables are non-negative integral-valued. 

The following result immediately follows: 


Theorem 1.9. The sum S, = X, +... + X, of п (a fixed number) identi- 
cally and independently distributed random variables Y; has the L.T. 
equal to [/*(s)]^. f*(s) being the. L.T. of X. 

As an application we consider the following result: 


Theorem 1.10. The sum of k identical and independent negative expo- 
nential distributions with parameter A follows gamma distribution with 
parameters A, k. 
Let б» Ле 1, Dasol 2, ..., ks have densities 
f(x) = exp (—Ax),x>0 for alli. 
From Example 3(c), the L.T. of X; is 
Si*(s) +- As +A) 


and so ће L.T. of S = X, + X, +... + Хк is (As +A))*. 
But this is the L.T. of the gamma distribution with density f(x) given 
in (3.15) (Example 3(c)). Hence the theorem. A 

We сап easily obtain the mean and variance of the gamma distribu- 

tion as follows: 
E(S) = E( Xj) 2 ЕХ) = К/А 
var(S) = È var(X;) = К/А? (see (3.17) and (3.18)). 

We shall show that there is a close analogy between the (discrete) 
geometric distribution and the (continuous) negative exponential distri- 
bution (Sec. 4.4). Negative binomial distribution, which can be obtained 
as the sum of a number of independent geometric distributions (Example 
I(g)) has the continuous analogue in the gamma distribution which can 
be obtained as the sum of independent exponential distributions. 


Limiting form of Erlang distribution: As К — оо, Erlang distribution 
tends to a discrete distribution concentrated at 1/A. Let {£x} be a 
sequence of r.v.'s, Ex having Erlang distribution with density 
(AK) x! exp( — K 
Sik, k I(K) 
А k ү 
Its 1 Т. isgiven by f;* (s) -( — | — (143 
Buen by fe* (5) (; + Ak 3k E 
As k — co, f(s) > exp ( — s/A). (3.24) * 


AN) то. 


| 
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But exp(—s/A) is the L.T. of the discrete distribution concen- 
trated at а = 1/A (Example 3(a)). Therefore, from the continuity theorem 
it follows that, as k — co, the distribution of E; tends to that variable 
whose whole mass is concentrated at 1/A. 

This can also be seen from the fact that, as k — оо, the mode (k — 1)/kA 
of Бк moves to the right to 1/A and that as var (£x) = 1/kA? — 0, the 
whole mass of £ tends to concentrate at the single point 1/A. 

We thus have exponential distribution for k = 1 and the deterministic 
case fork — со. A suitable value of k (1 < k < oo) gives a good fit 
for many empirical distributions arising in practice. 

An idea of how the density varies as K increases can be seen from 
Fig: 1.3. 


1.0 


f(x) 


1 
2 


1 
À 


E 


x 


Fig. 1.3. The density function of Erlang distribution for some values of k 


C. Sum of a Random Number of Continuous Random Variables 

Iñ Sec. 1.1. we considered thesum Sy of a random number N of random 
variables Х;. Thzorem 1.3 is stated in terms of the p.g.f. of X; on the 
assumption that the .Y;'s are discrete variables (with non-negative integral 
values). Here we assume that the Аз are mutually independent and 
identically distributed variables which may be continuous as well. 

Replacing the p.g.f. P(s) of X; by its L.T. /*(s), we can get an analogous 
result, This may be stated as follows: 


Theorem 1.11. 1f Y; are identically and independently distributed random 
variables having for their L.T.'s 

f*(s) = Efexp ( 5X2) for all i (3.25) 
and if N (independent of Y;’s) is a random variable having p.g.f. 


G(s) = Z Pr {N =n} 5" = È g, 5", 


then the L.T. F(s) of the sum Sy = X, t... + Xv, 
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is given by F(s) = G(f *(s)). (3.26) 
The proof is identical with that of Theorem 1.3. It also follows that 
E {Sy} = E{Xj} E {N}. (3.27) 


It is to be noted that Sy is a continuous random variable. | 
Corresponding to Лу == Рг {Sy =j} we shall have density function f(x) 
of Sy, so that 


f(x) dx = Pr (x < Sy < x + dx} = 5 рг {М = папіх< 5, < х dx} 
n=0 


(3.28) 
= 5 Pr{N =n} Pr(x < S, < х + dx] 


n=0 


= Eg, Pr{x < S,« х -+ dx). (3.29) 


Example 3(g). · Random (geometric) sum of exponential variables: 
If X5, i= 0, 2,... have negative exponential distribution each with mean 
1/A and Sy = X, +...+ Ху, where N has geometric distribution given by 
Pr {N =n} = pq", n = 1,2,... then Sy has negative exponential distri- 
bution with mean 1/Ap. 

The p.g.f. of N is G(s) = ps/(1 — qs) 
and the L.T. of the r.v: X; is f*(s) = A/(s + A). 
From the Theorem 1.11, it follows that the sum Sy = X, +... -- Xw has 
L.T. equal to G(f*(s)). Replacing sin G(s) by f*(s) = A/(s + A), we have 


Bros РАКЕЛ) pA 

сич = Р (2.30) 
but this is the L.T. of the exponential -distribution with mean 1/Ap. 
Hence the probability element f(x) dx of Sy is given by 


f(x) dx =Pr (x < Sy < х + dx} = Ap exp ( — Арх) dx. (3.31) 


Example 3(h). Let X; have exponential distribution with mean 1/а and 
let N have Poisson distribution with mean c. We have f *(s) = L.T. of 
X,=aj(s + a), and С(5)=е“@—1) and hence the L.T. of Sy X,--... Ху 
is G(F*(s)) 


— exp (<(; T A 1)), replacing 5 by f *(s) = a/(s +- a) 


= exp ( — cs/(s + a)). 
What is the probability density function of Sy? 


1.4 Difference Equations 


Let f(n) bea function defined only for non-negative integral values of the 
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‘argument п. The first difference of f(n) is defined by the increment 
of f(n) and is denoted by A f(n), i.e. 


Af (n) — f (n+ 1) — f(n). (4.1) 
The second and the higher differences are defined by 
AFH f(n) = A(A*f(n), k > 0. 
It can be verified that 
AIS (n) = A*(A f(n)) = АҢ / (п + 1) —f(n) 
= A*f(n + 1) — AF f(n), k > 0. (4.2) 


We denote /(n + 1) by Ef(n) (E is here the displacement operator and 
is not to be confused with the symbol E for the expectation of a 
random variable): then 


Af (a) = Ef(n)— f(n) = (E — 1)/ (п). 
Symbolically, A=E—-1 о E=1+A4; we have 
E* f (n) = f (n 4 К). (4 3) 
By a difference equation, we mean an equation involving a function 
evaluated at arguments which differ by any of a fixed number of values. 
Examples of such equations are: 


f(n 2)—f(n4- 1) —f() —0 
(А + р) = Unti Рањ F Na п 22 1 
Рп — QPn-1— Р(1 — Ра-1) = 0 (4.4) 
у= a(l + ЮУ аву, +1 
GgUx + а +... + акШхук = g(x). 

In the first equation the argument runs from n to (л + 2); this is a 


difference equation of order (n + 2)—n=2. The last equation is of 
order (x А) — x — Kk. The first two equations are homogeneous while 


the last three are not. 
The equation f (n + 2) — f(n + 1) — f(n) = 0 may be written as 
E?f(n) — Ef(n) —f(n) = 0, or r(E)f (n) —0, 
where r(E) = (Е — E — 1). 
A difference equation may be written as a function of the operator E 
in the same way that a differential equation can be written as a function 
of the operator D (s x) There is indeed a close similarity between 


the theory of differential equations and that of difference equations. 
Given a function /(л) = Са", we can easily get the difference equa- 

tion f(n + 1) — af(n) =0. We shall be generally concerned with the 

inverse problem: given а difference equation, to find the function which 


Satisfies the given difference equation. 
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GENERAL SOLUTION OF A HOMOGENEOUS DIFFERENCE 
EQUATION WITH CONSTANT COEFFICIENTS 


Consider a homogeneous difference equation of order two 
/(п +2) жап ++ 1) + Ь/(п) =0 (4.5) 
ог r(E)f(n) = 0, where r(E) = E? + aE + b. 

Consider a trial solution f (т) = m" (т 5 0); 

then Ef(n) = m"*!, E?f(n) = m"** so that (4.5) gives 
(т? + am + b) = 0. (4.6) 

This equation r(m)=0 is called the characteristic (or auxiliary or 
secular) equation of the difference equation (4.5). 

(1) If the roots m, m, of theequation (4.6) are distinct, then mm" and 
т," are two solutions of the equation (4.5). Since the equation is homo- 
geneous суту" and с.т," will also satisfy the equation and so also 
will their sum. In other words, if m, m, are the distinct roots of (4.6) 
and c,, c, are arbitrary constants, then 

f (n) = eam, + em, (4.7) 
will be the general solution of the equation (4.5). | 
(ii) In case the roots n1, т, are equal, the general solution is given by 

f(n) = (су + соп)т,". (4.8) 
(iii) In case the roots are complex, then we сап write 

m, = r(cos0 + i sin0) and m, = r(cos0 — isin 0) 

and the general solution (4.7) will take the form , 

f(n) = Ar" cos (n9 + B), (4.9) 
where A, B are arbitrary constants. 

The above arguments can be'carried over to homogeneous difference 
equations of any order k. We may state as follows the rule for the 
solution of the homogeneous difference equations: 

f (n) + afin + I+... ак/(п ++ k) = 0 (4.10) 
or, r(E)f (0) = 0. 
First, find the roots of the characteristic equation 
тт) = 1 4 am -- .. . + ат = 0. (4.11) 

Suppose that ту, т,,..., my are the real and distinct roots of (4.11), 

then the general solution of (4.10) will be given by 
f(n) = сүтү" + emp +... ++ cmt. (4.12) 


In case some of the real roots are equal, say m, = т,, and others 
distinct then the solution will be 


f (n) = (с, + enm, + ст," +... H eem. (4.13) 
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It is to be noted that the general solution of an equation of order К 
will, in general, contain k arbitrary constants с;, с,,..., ск. These can be 
obtained when & initial (or boundary) conditions are also given. 

General solution of a general linear equation r(E) f(n) = g(n) is given 
by u + v, where wis a solution of the homogerteousequation (Е) f(n) = 0 
and v isa particular solution of the complete equation. Methods of 
obtaining this particular solution depend largely on the form of the 
function g(n). } 

There are other methods also for solving a homogeneous difference 
équation. Some of these, which could be used to solve certain types of 
non-homogeneous equations, are discussed in the next section. 


Example 4(a). Fibonacci numbers are given by 

А =0, Л = 1, fa =fr + fa- (п 2 2). 
The difference equation fa — f;-, r/,;-, has the characteristic equation 
m? = m + 1, whose roots are n, = (1 i \/ 5)/2 and my = (1 — y 5)/2. 
Hence fa = сүтү” -+ cam”. Solving for c, and c, by using / = 0, Л = 1, 
we get c, = —c, = 1/\/ 5, so that 


f, = A VS 23" — (1 — v/5)2yyiv 5. 
1.5 Difference Equations in Probability Theory 


Many problems in probability theory can be put down in terms of a 
difference equation involving the unknown probability. Such problems 
can then be solved completely by making use of the initial conditions 
supplied by the problem itself. This method is sometimes found to be 
very powerful. 


Example 5(a). To solve the difference equation 
Un = (Us, + Un-y), 1 п<а-—-1 
having initial conditions 
U, —1 and U,=0, 
The difference equation has the characteristic equation 4m? — m + à = 0 
having two equal roots m -= 1, 1. The solution of the difference equation 
is 
Un = (e, + ejm) = с +- en. 
The initial conditions give 
l = U= Cp 0 Us =c; + са, whence c, = — l/a. 
The solution of the difference equation is thus 
U,—1 + (—1/a)n = 1 — nfa. 
(The equation arises inthe classical ruin problem, U, being the pro- 
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bability of gambler’s ruin in the symmetric case p = q=}3.) In the 
study of stochastic processes, we frequently come across equations which 
are non-homogeneous and which have coefficients that are not constants 
but functions of the argument. The method of induction and the method 
of generating function could be used in such cases. These methods are 
illustrated by an example. 


Example 5(b) The probability function 
ps» = Pr {№ = п), п 0238... 
of a random variable N satisfies the difference equations: 
Puy — (1 + а) Pn + ара- = 0, п 2.1 (5.1) 
—p,+apy=90 (0 «a < 1). (5.2) 
Since p,s are probabilities of the random variable N, we have also the 


condition 


X p=: (53) 


n-0 
We now proceed to find p, i.e. to solve the difference equation (5.1) 
given (5.2) and (5.3). 


METHOD OF CHARACTERISTIC EQUATION | 
The second order difference equation (5.1) which can also be written as 


Puma — (1 a) Dua + арһ = 0, п > 0 (5.4) 


has the characteristic equation т? — (1 -- a) т -- a = 0, whose roots 
are m==1 and m = а (< 1). So the equation (5.1) has the solution 


Ра = А(1)" + В(а)" = А + Ва", п > 0. (5.5) 
We have from (5.4) and (5.2) 
Pı = А -+ Ва = ap, 
po А + B. (5.6) 
Solving in terms of py, we get A = 0, В = p, so that 


Pn = Po". (5,7) 
From (5.3) we get 


1 =2Ē2 р, = È pa" = р/(1—а) or pp=1—a. 
Hence the solution is ý 
Pa= (1 — а) а", п>: 0. (5.8) 
METHOD OF INDUCTION 
The equation р; — (1 + a) р, + ар,_; = 0 can be written as 


Panti — 4Pn = р, — apg-,, n > 1. 
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Putting (n — 1) for n, we get 
Pn — арк-ү = Pr- — ар,-, 
and so on. Hence 


Ра+у — ра = Pn — OPn-y = Pa-y — ар- = ... 


=P, — ару, n> 0. (5.9) 
But from (5.2) Рі — ap, =0. 
Hence 
Рп = OPn- = а(арь-,) =. . . = атр, 
and 


i pac gives p, — 1 — a. so that p, = (1 — а) a", n > 0. 


METHOD or GeNERATING FUNCTIONS 
Let 


P(syes EUIS і (5.10) 


be the p.g.f. of the distribution. 
Multiplying equation (5.1) by s"(n > 1) and adding over n = 1,2, ..., 
we get 


5 Pav - (1.8) È past a È pas-0 (5.11) 
or, (1/8) [P(s) — Pa — Ра] — (1 + a) [P(s) — po] + asP(s) = 0 
or,  [l/s— (1+ a) + as] F3) 0 + p, — (1+ др 


(1 —5X1— as) рз) = 0 SN 
sS 5 


ог, 
Непсе 
Р(ѕ) = 1 Bh (1 — as)! = p, (1 + as + as? +...) 
so that 
Рп = coefficient of s" in P(s) (5.12) 


= pu. 
Finding ро as before from (5.3), we get 
Pr=(1—a)a, nzo. 
The method of Laplace transforms can also be used to solve a difference 
equation (Smith Ch. 9). 


Example 5(c). To solve the difference equation 

иһ = 40-1 + PC — n) n > 1,p+q=1, (5.13) 
given the initial condition m= 1. It is a nonhomogeneous equation of 
order 1. Only one initial condition is needed to solve it completely. 
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Let A(s)= Ў uns" be the generating function of {un} given by (5.13). 
n=0 
Multiplying (5.13) by s” and adding over л = 1,2, ..., we get 


X иһ5" —(q—p)Z uw s" +p Es 
n=l n=l n=l 


or, (AG) — 1) = (a — P)sA()-- p* 1—À 
Or, {1 — (q —p) 5} As) = cs +1 (since иу = 1) (5.14) 
| ps 1 
oh 49) = (=a — р) (70-5 
e aT 1 4 1 
=i lis 1 Aem "I= Q= PpS 
(decomposing into partial fractions) 
= ${(1 — s)" + [1 —(@—p)s]-}. 
Thus 


u„= coefficient of s" in A(s)= $ {1 + (a — p). 


(Here и, gives the probability that n Bernoulli trials result іп an even 
number of successes. {un} is not a probability distribution.) k 

The method of generating functions is very powerful. Itcan also be 
used when the coefficients of the equation are not constants, but are 
functions of the argument. 


1.6 Differential-Difference Equations 
Suppose that u,(t), = 1, 2,..., is a function of t having a derivative 


d m(t) = w (0). 


An equation involving un (f) and u,(t), un4(t) etc. is called a differential- 
difference (or recurrence differential) equation. For example, 


py (t) = — A Ust) — Par (0 


is a differential-difference equation (see Bellman and Cooke fora detailed 
account of this topic). j 
The existence of unique solution will depend on suitable initial condi- 
tions which are to be determined carefully. The solution of such equations 
can sometimes be found easily by the method of induction (see Example 
5(b). Besides, there are two powerful tools for the solution of such an 
equation: these are generating functions and Laplace transforms. By 
using generating functions, a differential-difference equation can often be 
reduced to an ordinary differential equation, while the use of Laplace 
transforms often enables one to reduce a differential-difference equation 


3 
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to a difference equation. A combination of the two tools sometimes 
facilitates, the solution to a differential-difference equation. We illustrate 
the use of these tools by some examples. 


Example 6(a). Solve the differential-difference equation 


и) usn t DROP icd о (6.1) 
given the initial conditions 
u(t)—11:20 (6.2) 
and 
uy(0) — 1, u,(0) = 0 for n #0 (6.2a) 
Let 
GD = X ut) (6.3) 
n=0 


be the generating function of the sequence {u,(f)}; assuming the validity 
of term by term differentiation, we get 


G. © , vow 
ar c X wet arva X ase 
дї п=0 п=1 


= B и'(1)5”, since u'(t) — 0 from (6.2). 
n=l 
Multiplying (6.1) by s” and adding over for n = 1, 2,..., we get 
z US (Est = Z Una (1)5" 
nel nel 


әс i 
or, or = sG, 


which is a partial differential equation. The solution is 
G(s, t) = A exp (st), 


and putting / = 0, we get G(s, 0) = A. 
Again, G(s, 0) — 5 Un(0)s" — uj(0) „рә us(0)s^ =1, from (6.2а). 


Hence A- 1 and G(s, t) = exp(st). 


Thus, ] 
u,(t) = coefficient of s" in G(s, t) 
coefficient of s" in exp (st) 
өй, л =0,1,2,.... 
n! 
Example 6(b). Consider the equations 
P(t) = — рї) — Preah n1, 320 (6.4) 


(6.5) 


п) = — Ар(4) 
P(t) Pol (6.6) 


with the initial conditions Рл(0) = 0 for л #0, po(0) = 1. 
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Let 


L{p,(t)} = Í exp (— st)pa(t)dt = f(s) (6.7) 


0 
be the L.T. of p,(t); then 
ІХ ps (0) = S (5) — p«(0), by equation (2.6) 
-—Ssfs,nz1l1 
Hence taking the L.T. of both sides of (6.4), we have 
S fals) = — Af S) — Ла (8)] (6.8) 


o, M= (369a > 1, 
which is a difference equation in f,. Its solution is given by 


DEOS (ж). п> 1. (6.9) 


By taking LT. of (6.5) and using the initial condition р,(0) — 1, we get 
Sfo(s) — 1 = — à f(s) 
or, So(s) = 1f(s + А). (6.10) 


The unknown constant с in (6.9) can Be: obtained by putting п= 1 апі 
using (6.10). We have 


Fs) = e= Ws +A). 


` Hence 
Anci 
fias) а’ п 2 1 
ог, fs) = er n>0. (6.11) 


Now,'p,(t) can then be obtained by finding the inverse Laplace trans- 
form of f,(s). From Example 2(b) we find that f,(s) is the L. T. of 


exp (— Ar)(At)" 
n! 
Hence 
exp (— At)" 
n! 2 


р!) = n= 0,1,2... (6.12) 


The solutions of these equations governing Poisson processes will be 
obtained by an alternative method of generating function in Sec. 4.1 
(Theorem 4. 1). 


Example 6(c) Consider the equations 
py (t) = — (a + B) pat) + арһ-({) + Бр), nz 1 _ (6.13) 
po (t) = — ард!) + bp,(t) (6.14) 


INTRODUCTION 33 


with initial conditions p,(0) — 0, л #0 and p,(0) — 1. (6.15) 
Let 


F,(s) = J exp(-—st)ps(t)dt bethe L.T. of p,(t). 
0 
By taking L.T. of both sides of (6.13), we get the difference equation 
sFy(s) — p.(0) = — (a + b)Fa(s):+ а, (5) + РР,41(5), n > 1 


a 
at+b+s 


or, F,(s) = Fya(s) + At vs Е,.1(5). (6:16) 


Similarly from (6. 14), we get 
Роз) — py(0) = — aF (s) + bF(s) 


b 
FEE D a s OF (6.17) 


The L.T. of p,(t) for n > I satisfies a homogeneous difference equation 
of order two. Its characteristic equation is 


or, Fs) = 


ee ee 
or, bm? — (a 4- b+ s)m 4- a = 0. (6.18) 
If m,, m, are the two roots of this equation, then the solution of (6.16) is 
F,(s) = Am," + Вт", mz 1, (6.19) 
where A, B are two constants. By putting the value of F,(s), we get F,(s) 
from (6.17). 


Thus we get F,(s) the L.T. of рді) (п > 0), whence p,(t) can be 
determined. This technique due to Conolly (1958) leads to the solution 


of the M/M/1 queueing problem (see Sec 10.3). 
1.7 Matrix Analysis 


A. Eigenvalues and Eigenvectors 


Let A = (ау), where а; are complex numbers, be an m x m matrix and 
let Zm denote the unit matrix of order m. Let x denote a column vector 


т u 
| às | and x’ the row vector (ху... Xm); thus x — (x... Xm). 
Xm 

Definitions. The roots £j, i= 1,2, .... m, of the equation det (4— 1) = 0 
are called the eigenvalues (or spectral values, latent or characteristic roots) 
of the matrix A. The equation det (4 — 17) —0 which is of degree m 


over t is called the characteristic equation of A. ў 

The set T = (t, i= 1.2, -- .,m} of eigenvalues is called the spectrum 
of A, and p(A) be max {4 ---, fm} is called the spectral radius of A. 

A non-zero column vector x which satisfies the equations (А—11)х=0 


3(45-95/1982) 
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is called a right eigenvector (latent or characteristic vector) of A corres- 
ponding то (or associated with) the eigenvalue +. 

A non-zero row vector y' which satisfies the equation y’(A — t,/) 0 is 
called a /eft eigenvector of A corresponding to ty. 

Now (4 — 11). х - 0 has non-zero solution x iff det(A — 11) = 0, 
that is, iff г; is an eigenvalue of А. Ап alternative way of defining eigen- 
values and cigenvectors is as follows: 


A number 2 is called an eigenvalue of the matrix A if there exists a 
non-zero (column) vector x such that Ax — /x; the non-zero vector x is 
called the right eigenvector of A corresponding to t. 

The equation Ax = /x being homogeneous, the eigenvector X cannot 
be unique. 

It may be noted that the equation y' (A — 11) = 0 is equivalent to 
(A' — 11) y--0. This can be immediately shown by using the property 
that for two matrices Y, Y, 


(XY)! = Y'x. 


We state here some simple properties of eigenvalues and eigenvectors 
(the eigenvalues t;, i — 1,..., m of A need not necessarily be distinct): 


l. Det A--Oiff А has a zero eigenvalue. 
2. The matrix A and its transverse A’ have the same eigenvalues. 
3. Different matrices may have the same eigenvalues. For example, 


k o and c have the same eigenval 
0 2 n (бга have the same еіре ues. 


4. Ifa matrix A is of the form 


A 0 
i-i t) 
By Ass 


, H " H " H 
where 411, As. are square matrices, then is an eigenvalue of 4 iff it 
is an eigenvalue of at least one of the matrices Ау, Ago. 


The sum of the diagonal elements (Le. Уан) of A of order mis known 


as the trace of A and is denoted by tr (A). Det (A) and tr (A) are 
expressible in terms of the eigenvalues t; as follows: 


т m 

tr (A) = Zt, det A= IH h. | 

i=l ї=1 [ 

6. If f(A) is а polynomial (scalar) in A, then the eigenvalues of f(A) are 
fü), i=1,...,m. In particular, the eigenvalues of А" аге (г у", 
HN PA 

7. The eigenvectors form a biorthogonal system. Let ti, t; be two 


distinct eigenvalues with x; x; and у, Y; as the corresponding 
eigenvectors. 


We have 
Ax;= tx, and y,'A = уу 
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and (уух) = (гуу) Xi = QA) Xi 
= yj (Ах) = ух = ti(y;'xi). 


Since t;  t;, equality holds iff y;'x, = 0. 
B. Nonnegative Matrices 


We shall now confine ourselves to square matrices with real elements. 
A matrix A is nonnegative (40) if all the elements of A are non- 
negative, ie. 42-0 if aj; 20, and Ais positive (A > 0) if ау > 0 
formali j= 1, 2,..., m. 


Definition. A square matrix in which each row and each column has 
exactly one entry with value unity, and all others zero is called a permu- 
tation matrix. 

Note that there are m ! permutation matrices of order т. An т х m 
matrix A is said to be reducible if there exists a permutation matrix P 


such that 
rem Ay, a) 
PAP =( Ды 


where 4,1, As are squares matrices of order r and m — r respectively. 

A is irreducible if no such permutation matrix P exists. 

It is to be noted that a transformation of the matrix A of the form 
Р' АР has simply the effect of rearranging the elements of A. The 
reducibility of a matrix depends on the disposition of the zero and non- 
zero elements of A. A matrix A with all positive real elements is neces- 
sarily irreducible. Further, if the matrix is irreducible, all the off-diagonal 
entries of any row or column cannot vanish. We state below an elegant 
result concerning nonnegative matrices due to Perron-Frobenius. 


Theorem 1.12. Let A be anonnegative and irreducible matrix. Then 
(1) A has a positive eigenvalue, r, equal to its spectral radius p(A), 
(2) risa simple eigenvalue of А, i.e. r is a simple root of the charac- 

teristic equation, 
(3) there is a positive right eigenvector associated with r, and 
(4) rincreases when any entry of A increases. 


Perron obtained the result in 1907 for positive matrices; it was later 
extended by Frobenius in 1912 to nonnegative matrices. If A is positive, 
(when it is necessarily irreducible) Perron showed that there is only 
one eigenvalue of modulus г. If A is nonnegative and irreducible, 
then the number of eigenvalues having modulus equal to r (= p(4)) is h, 


where Л > 1. 
Definition. An nonnegative irreducible matrix is called primitive, if 


h=- |, and imprimitive (or cyclic) of index h, if h > 1. 
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A positive matrix is necessarily primitive. 

A nonnegative irreducible matrix A may be primitive or imprimitive. 
Suppose that A is cyclic of index h and t;, i — 1, 2,.. .,/ are the h eigen- 
values of modulus r, i.e. t; = r = p(A), then t; = wr, i — 1,..., h, where 
wrs are the Л distinct roots of unity. 


C. Spectral Representation of a Matrix 


Consider an m x m matrix A with m distinct non-zero eigenvalues f; and 
Xi, yi as the right and left eigenvectors, i — 1,2,..., m,of A. Let D= 
(dij), di; ~ 0,1 Æj, di; = ti bea diagonal matrix and let X = (95,5, X5) and 
Y = (yy -+-» Yn) be the matrices formed with the right and left eigenvectors 
respectively of 4. From y;//4 — t;y; and Ax; = t;x;, we get Y'A — DY' 
and AX= XD respectively. Now suppose X-! exists, then AX — XD 
implies ХААХ)! = X-1(YD)X-!, or Х-1А— DX-, Thus the rows of 
X~ are the left eigenvectors of A. 


Particular case: Suppose that X-! = Y'; this implies that YY’ — 7, or 
Y'X = І. We may thus choose suitable multiplicative factors to get 
у'х = 1. Since y'x -- 0, i+ j, we may write уух, = 3j. From AX -- XD 
we get A = XDX-! = XDY'. Further A? = (XDX-!) (YDX-1) = YD?X ! 
= XD*Y' and A" = XD"Y' (for any positive integer n > 1), where D^ isa 
diagonal matrix with elements ti", i = 1,2,..., т, along the main diagonal, 
the off-diagonal elements being all zero. On expanding, we get 


т 
А"= 2 tjxjy;'. 
a" Jal 
General case: , Suppose that the eigenvectors x and y’ are arbitrarily 
chosen; then we can write 


yx; =B;i/e;, or с; = (уух) 
A" — X tieixiy/' n 1. 
j=l 


The products c;x;y; ~ B; are matrices (called constituent matrices) and 
the above gives an expression of A” as a sum of m such matrices. 


We immediately find the following properties of constituent matrices : 
(1) B;B;—0, foris j: 


(ii) Bj — В, (Вуз are idempotent); 
m 
and (iii) X B,—L. 
ja 


The representation 4-- YDY' is known as the spectral (or 
canonical) representation of A. 
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Example 7(a). Consider the matrix 


2 1 
A= 
(34) 

The solutions of the equation det (4 — tI) ==0 are 1 and 5, іе. г, = 1 
t= 5 аге the eigenvalues. The right eigenvector x, (where x,' = (x à X, D 
corresponding to t,=1 is given asthe solution of Ax, = x, es y 

2х1 + ху = Xn and 3x + 4х = хуу. 
Solving we get, - 

ху = (хь Хз) = (1, —1). 
Corresponding to t =5, to right eigenvector X, (where x," = (Xa %g9)) 
is given as the solutions of Ax, = 5x,, i.e. ‘ i 
2x F X, 5x4 and 3х, + 4х = 5, 
— n 22 21 21 22 Хаз; 
х; = (Xa Хаз) = (1,3). 

Тһе left eigenvectors y;' = (у, Yiz) can be similarly obtained as solutions 
of у А = tyi, і —1,2. We get 


y! = (ль з) =(- 31) 
and 
У, = (72 а) = (1, 1). 


с = Шух) =—ф = 1/(у; Хх„) = 4, 
"eL 1 5 1 
хуу, =N 3 21) х,у, =(, 3); 


2 
A= È nB, A= È tr В (r> 1), 
tel i=l 


Thus, 


where B, = cixiyi’. 
Verify that (i) x, y are biorthogonal, 


2 
(ii) BP = В, BB;=0, TAL È B= hy 
A 


D. Stochastic Matrix 


Definition. A square matrix with nonnegative elements and unit row 

sums is called a stochastic (Markov) matrix. It is doubly stochastic, if, in 

addition, it has unit column sums. 

P — (pij) is stochastic iff pj > 0, Х pi; = 1 for all i, j; 
i 


P is doubly stochastic iff ру > 0, z pij = 1, 2р) = 1 for all i, j. 


If P isa stochastic matrix, 50 also is Р", n being a positive integer. 
Stochastic matrices (which are nonnegative) have some very interesting 
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properties which are useful in the study of Markoy chains. Some of 
these are given below. 


Theorem 1.13. The matrix D is stochastic iff it has an eigenvalue 1 and 
the corresponding right eigenvector x is a column vector with all the 
elements equal to unity, i.e. х' = (1, 1,..., 1). Further p(P) = 1, i.e. the 
spectral radius of a stochastic matrix is 1. 

If the matrix P is irreducible, then there can be one and only one 
eigenvalue equal to 1 (Theorem 1.12). However, there can be h(> 1) 
eigenvalues of modulus 1 if P iscyclic of order h. If itis primitive (Ir T), 
then there can be no other eigenvalue of modulus 1. 
that lim P^ 5 0. 


ne 


From this we get 


Theorem 1.14. Suppose that P is an irreducible stochastic matrix. Then 
lim P" exists iff P is primitive. 
п» о 


Remark: The hypothesis that Р is irreducible is not necessary; the limit 
P" exists when P is a primitive stochastic matrix. 


These properties of stochastic matrices will be u 


sed in the study of 
Markov chains (Sec. 3.4). - 


EXERCISES 


1.1. (a) Find the generating function of the Fibonacci numbers {fna} 
(defined in Example 4(a)). 


(b) Let Х be a random variable Cenoting the number of tosses 
required to get two consecutive heads when a fair coin is 
tossed. Show that the p.g.f. of Yis (88/4) (1 — 5/2 — (sJ2)5)-1. 

Let Y have the distribution of the geometric form (modified 
gcometric or decapitated geometric) given by. 
Pr(Y =k} = q*-1p5, k= 1,2,3,.... 
Show that the p.g.f. of Y is Ps/(1 — sq), and that 
E(Y) = 1/р and var (Y) = q/p*. 


1.2. 


(The number of trials Y upto and including the first success ina 
sequence of Bernoulli trials with probability p of success has the 
above distribution; again Y-- Х + 1, where Y has geometric 
distribution considered in Example 1(с).) 

1.3. Let X be a random variable such that 


Pr(Y = К} = py, Pr(X > К} == ii pis Ks 0. 


If P(s)= È рык and O(s)— X gp sh, 
k=0 к=) 
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show that (1 —s) Q(s) -1— P(s) and that E(X) = Q(1). 

1.4. Find the mean and variance of the negative binomial (NB) distri- 
bution considered in Example l(g. Obtain the p.g.f. of the sum 
of m 11.4. NB variables and find its mean'and variance. 

1.5. (a) Let X have a zero-truncated (or decapitated) Poisson distribu- 

tion with zero class missing, i.e. 


йе. -1 zk 
pk — Pr {X= k} — (€ m S pega n 


Show that the p.g.f. of Y is given by P(s) = (оа — 1)-1 (eas — 1). 
Verify that P(1)— i рк = 1; show that £(X) = аев/(ев — 1). 
-1 


(b) If Y, Y are two independent zero-truncated Poisson variates, 
find the p.g.f. of the sum (Х+ Y) and the distribution 
Pr {X + Y =n}, n =2, 3,... when (i) X, Y are identical with 
the same parameter a, (ii) X, Y have parameters а, а, 
(see Medhi, Metrika, 20, 215-218 (1973)). 
(c) Find the p.g.f. of the sum S, = X, +...-+ X, of n independent 
and identical zero-truncated Poisson distributions. Find E(S,) 
and Pr(S, =m}, m=n,n+1,+2,.... - 
1.6. Let Sy =X, +...+ Ху, where X, are i.i.d. random variables 
and N is a random variable independent of Хз. Show that 


‚ var (Sy) = E{N} var (Xj) + var (N) {E(X}. 


and that ) 3 (н) 

var(Sy) _ уаг(л1 Ex) Yt 

ES.) - EQ) 5209 ЕЛ)” 
Show that for a compound Poisson or compound geometric 
distribution, 


var (Sy) 5, y var (X) 
GS ^ “EOD 
where k is simply related to the compounding parameter. 
1.7. Let Sy — X, + X, +... + Ху, where N has Poisson distribution 
with mean a. \ и 
(a) If X; have i.i.d. geometric distribution with mean g/p, find 
(i) the p.g f. of Sy, and (ii) E(Sy) and var (Sy). я 
(b) If X, have i.i.d. Bernoulli distributions with Pr (X, = 1) = р. 
and Pr {Ху = 0) = 1 — p =q, show that Sy has Poisson dis- 
tribution with mean ap. Show that the joint distribution of 
Sy and N has probability mass function 
е-ад"ру"-> 
Рг {N = A, КУЯ =i} = = y)! 
for nz20,1,2,... Y* Oily ape ot: 
and that the conditional distribution of N given Sy == y has 


probability mass function 
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1.8. 


1.9. 
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е-% (а4)"-> 
(n — у)! 
Show that cov (N, Sy) — ap. 


Рг {N=n|Sy = у} = ,nzy. 


(c) If X, have i.i.d.Poisson distributions with mean b, show 
that the joint distribution of Sy, and N has mass function 
exp {— (a + bn)) a" (nb 
n! y! 


Pr {N =n, Sy = у} = 


for n=0,1,2,...,y —0, ],.... 


Show that E(Sy) = ab and var (Sy) = ab(1 + Б). 
(Accident models have been constructed and fitted to actual 
data involving traffic accidents (Leiter and Hamdan, Int. Stat. 
Rev. 41(1973) 87-100) taking N as the number of accidents; Y, 
(in (b)) as the indicator variable indicating whether the ith 
accident results in a fatality or not; X; (in (c)) as the number 
of persons fatally injured in the ith accident.) 
Let Z = X,+...+Xy, where {N} has zero-truncated Poisson 
distribution (defined in 5(a) above) (N is independent of Ху) and 
X, have i.i.d. exponential distribution with mean 1/A. Show 
that the L.T. f*(s) of Z is 


f*(5) = {exp (M/A + в) — (es — 1), 
E(Z) = ae*/(A(e* — 1). 
It may be further verified that 


and that 


£0 = 5629 (2) һ evaa 
is the probability density function of Sy. (Show that L.T. of 
(t) is f*(s).) 
(1, (х) is the modified Bessel function of order r(> — 1) given by 

ed 1 

1, (x) mE иц кк KEP ET De), 
(The above result can be used in Queueing theory in a model 
of phase-type service, where the r.v. N denotes the number of 
phases associated with a customer's service, and the i i.d. r.v. X; 
denotes the duration of service time at the ith phase.) , 
The Bessel function density f(t) is quite similar, for the value of 
the parameter a of 10 or over, to the gamma density het). 
However f(t) gives better fit to alarge number of empirical dis- 
tributions (see Gaver Opns. Res. 2 (1954) 147). 
Suppose that X;, i= 1, 2 are two independent geometric variables 
with Pr (X; = k} = pq;, i = 1, 2, k — 0, 1, 2,.... Find the bivariate 
generating function P(s,, s,) of the pair (X, Х,) and from the form 
of the generating function for the sum S — X, + X,, verify that 


INTRODUCTION 3 2] 


k 
Pr {S =k} = 7 AUDIOS ар eo Ovals) eee 
1.10. Find the Laplace transforms of : 
(i) sinat; (ii) a exp (— 2t) ; (ii) (1° + 1); (iv) 13 exp (—3t); 
(у) e~ cos 2t; (vi) t%e%(n >— 1); 


{exp (—at)}(at)" 3 
) арсар (n—0,1,2,...); (viii) E 


(ix) VEDI); (x) {exp C7 t— DH(Q)EQVID; 
(uj ш Л) ема), 

111. If Цу) = FG) 
show that 


(vii 


O UK = -— Fs), 
(i) дии) =(— D* F'(), 
(Ui) — L(f'(0) = s? Fs) — sf(0) — 770). 
(f'(t) denotes differentiation with respect to г and F'(s) denotes 
differentiation with respect to s). 
(The generalisations of (i) and (ii) for higher powers oft and of 
(iii) for higher derivatives can also be obtained.) 
1.12, The non-negative valued random variable X is said to have à 
modified exponential distribution with an non-zero probability mass 
at ‘0’, if its distribution function is 
F(x) = Pr(X < x} = 1 —k exp (— Ах), 
x>0; А>0, Of k= 1. 
Find (i) the frequency density of X, (ii) 2А), and (iii) L.T. of X. 
1.13, Show that the sum of two independent gamma variables having 
parameters A, k and A, / is a gamma Variable with parameters 


A k+l 
1.14, Find the Laplace transform of the variable X (having triangular 


distribution) with density function 
Дх) = х, 0<х<1 
=2— x, ] seus. 


that the triangular distribution can be obtained as the 


Hence show р 
m variables each with a rectangular 


sum of two indépendent rando 
distribution in (0, 1). 
1.15, Solve the difference equations 


(i) fn--2-—/f0-9. 
(ii) Unto + 2Un t Un =0, 
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Qi) fin + 3) – 400 + 2) + Sfin + 1) — 2f(n) = 0. 
Solve the equation 
Un = рО, + qU,i-, | &n&a—1,0cp—1—a«1 
given that 
Uy = 1, Ua=9, (рт4). 
By taking the limit of U, as p/g—1, find the solution of 
Un = $ (Una, + Un) (see Example 5(a)). 


1.17. If pa(t), п = 0, 1, 2... represents a probability distribution and 


1.18. 


1.19. 


1.20. 


SAS) is the L.T. of p,(t), then show that 
X f(s) = s. 
nen 


Apply the method of Laplace transform (employed in Example 6(b)) 
to solve the equation of Example 6(a), viz. (6.1) with the given 
initial conditions (6.2) and (6.2a). 

Consider the differential-difference equations (6.13) and (6.14) with 


initial conditions (6.15) (see Example 6(c)). 
If 


G(z,t) = Х pz 


show that 
9G(z, t 
z 2 = (1 — 2) ((b — az)G(z, t) — bp,(t)}- 
If Pn*(s) = L{p,(t)} and G*(z, s) = L{G(z, t)}, 
then show that 
(> s) = 2001 — 2) poX(s) 
бї (дз) = sz— (1— z) (b — az)" 

(These relations occur in the study of transient solution of M/M/1 
queue (Bailey 1954)); see Sec 10 3.) 
Suppose that P,(t) satisfies the differential-difference equations 

Р, (1) = — (a + Б)Р (2) + aPn-y(t) + bPns,(t), n > 2 

P(t) = — (a + Б)Р,(ї) + БР) 

Р, (а) = bP,(t) 
with initial conditions P,(0) = 1, P,(0) = 0 for n Æ 1. 


Show that the generating function G(z, t) = b P,(t)z" satisfies 
n-0 


2 = = (1 — 2)(6 — ат){С( 2,1) — P(t)); 
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1.21. 


1.22. 


1.23. 


1.24. 


1.25. 


show that L.T. G*(z, s) of G(z, t) is given by 


usa 2-0 30 адр) 
G9) = s — (i — ab — az) 


(These relations lead to busy period distribution of A//A//1 queue 
Bailey (1954).) 
Find the eigenvalues and the left eigenvectors of the matrix 


22 5 0 
Р=| 2.5 0 soy jm 
0 1 0 


Examine whether the following matrices are primitive : 


NON itor 
(i) Б (ii) 1 
mM бк 0 0 
о ао. ND 
0 E оо 12 д2 
Е AR ale gunn “pe 0 0 1/2 12 
ii) Е n 107 0 ED 
iB. “Др oO. 0 iB i 0 6 
oor me o 
00: О 
ООО Оо ре / 
à "Y oo 


If the two m x m matrices P and Q are, stochastic, then show 
that the matrix 


(i) PQ is stochastic 
(ii) аР+(1— a)Q is stochastic, for 0 < a < 1. 


Is the product of two doubly stochastic matrices doubly stochastic? 
Probability vector : à vector x! = (Xy. . Xi. + +s Хт)» such that 


x; 20, X x; = 1 is called a probability vector. 
iei : 


Show that an m x m matrix M is stochastic iff x'M is a probabi- 
lity vector whenever x is a probability vector. (This is a char- 


acterization of stochastic matrices.) 
Prove that a doubly stochastic m x m matrix A may be written as 


A = XoP, г = l, otl, c, 2 0, Ўсе = 1, 


where P, are permutation matrices of order т. 
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CHAPTER 2 


STOCHASTIC PROCESSES 


2.1 Notion of Stochastic Processes 


Since the last century there have been marked changes in the appro- 
ach to scientific enquiries. There has been greater realisation that probab- 
ility (or non-deterministic) models are more realistic than deterministic 
models in many situations. Observations taken at different time points 
rather than those taken at a fixed period of time began to engage the 
attention of probabilists. This led to a new concept of indeterminism— 
indeterminism in dynamic studies. This has been called “dynamic inde- 
terminism” (Neyman (1960), traces the transitions). The period of 
dynamic indeterminism began roughly with the work of Mendel (1822- 
1884). The physicists played a leading role in the development of 
dynamic indeterminism. Many a phenomenon occurring in physical and 
life sciences are studied now not only as a random phenomenon but also 
as one changing with time or space. Similar considerations are also made 
in other areas, such as, social sciences, engineering and management 
and so on. The scope of applications of random variables which are 
functions of time or space or both has been on the increase. 

Families of random variables which are functions of say, time, are 
known as stochastic processes (or random processes, or random func- 
tions). A few simple examples are given as illustrations. 


Example 1(а). Consider a simple experiment like throwing a true die. 
(i) Suppose that X; is the outcome of the mth throw, n> 1. Then (X, 
n> 1) isa family of random variables such that for a distinct value of 
п (= 1,2,...), one gets a distinct random variable Х„. (X,, n > 1) consti- 
tutes a stochastic process, known as Bernoulli process. (ii) Suppose that 
X, is the number of sixes in the first л throws. For a distinct value of 
n-1,2,..,we get a distinct binomial variable Х,; {X,, n > 1) which 
gives a family of random variables is a stochastic process. (iii) Suppose 
that X, is the maximum number shown in the first 7 throws. Here 


{Xn n > 1} constitutes а stochastic process. 


Example 1(b). Consider that there are r cells and an infinitely large 
number of identical balls and that balls are thrown at random, one by 
one, into the cells, the bali thrown being equally likely to go into any 
one of the cells. Suppose that X, is the number of occupied cells after n 


throws. Then (X, n > 1) constitutes a stochastic process. 
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Example l(c) Consider a random event occurring in time, such as, 
number of telephone calls received at a switchboard. Suppose that X(t) is 
the random variable which represents the number of incoming calls in an 
interval (0,7) of duration і units. The number of calls within a fixed 
interval of specified duration, say, one unit of time, is a random vari- 


able X(1) and the family (X(t), t © T) constitutes a stochastic process 
(T -- [0. оо)). 


2.2 Specification of Stochastic Processes 


The set of possible values of an individual random variable X, (X(t)) 
of a stochastic process {Xn n > 1} (X(t), t © T) is known as its state space. 
The state space is discrete if it contains a finite or а denumerable infinity 
of points; otherwise, it is continuous. For example, if X, is the total 
number of sixes appearing in the first throws of a die, the set of possible 
values of Y,is the finite set of nonnegative integers 0, 1,...,п, Here 
the state space of X, is discrete. We can write X, —Y,--...--Y,, where 
Y; is a discrete г. v. denoting the outcome of the ith throw and У; = 1 
orÜ according as the ith throw shows six or not. Secondly, consider 
Х„ =Z,+...4+2Z,, where Z; isa continuous г. v. assuming values in 
[0, со). Неге the set of possible values of XY, is the interval [0, oo), 
and so the state space of Х, is continuous. 

In the above two examples we assume that the parameter n of X, is 
restricted to the nonnegative integers п = 0, 1, 2,.... We consider the 
state of the system at distinct time points n = 0, 1, 2,..., only. Here 
the word time is used in a wide sense. We note that in the first case 
considered above the “time л” implies throw number n. 

On the other hand, one can visualise a family of random variables 
{Xa t & T) (or (X(t), t © T} such that the state of the system is chara- 
acterized at every instant over a finite or infinite interval. The system 
is then defined for a continuous range of times and we say that we have 
a family of r.v. in continuous time. A stochastic process in continuous 
time may have either a discrete or a continuous state space. For example, 
suppose that X(r) gives the number of incoming calls at a switch- 
board in an interval (0, t). Here the state space of X(t) is discrete though 
X(t) is defined for a continuous range of time. We have a process in 
continuous time having a discrete state space. Suppose X(t) represents 
the maximum temperature at a particular place in (0, t), then the set of 
possible values of X(t) is continuous. Неге we have a system in conti- 
nuous time having a continuous state space. 

So far we have assumed that the values assumed by the r.v. Х„ 
(or X(t)) are one-dimensional, but the process (X,) (or {X(t)}) may be 
multi-dimensional. Consider X(t) = (X,(t), X,(t)), where X, represents 
the maximum and X, the minimum temperature at a place in an interval 
of time (0, г). We have here a two-dimensional stochastic process in 
continuous time having continuous state space. One can similarly have 
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multi-dimensiona! processes. A one-dimensional process сап be classified 
into one of the following four types of processes: 


il) Discrete time, discrete state space 

(ii) Discrete time, continuous state space 
(iii) Continuous time, discrete state space 
(iv) Continuous time, continuous state space. 


All the four types may be represented by {X(t), t є 7). In case of 
discrete time, the parameter generally used is п, i.e. the family is repre- 
sented by (X,, n = 0, 1, 2,...). In case of continuous time both the 
symbols (X, t © Т} and (X(t), t © T) (where T isa finite or infinite 
interval) are used. The parameter: is usually interpreted as time, 
though it may represent such characters as distance, length, thickness 
and so on. Some authors call the discrete parameter family a stochastic 
sequence, and the continuous parameter family a stochastic process. 

We shall use here the notation {X(t), ТЄ T) both in the cases of 
discrete and continuous parameters and shall specify 7, whenever 


necessary. 


Relationship 

In some of the cases, the r.v. Xn, і.е. members of the family {Х,} are 
mutually independent (as in Example 1 (a)), but more often they are 
not. We generally come across processes whose members are mutually 
dependent. The relationship among them is often of great importance. 

The nature of dependence could be infinitely varied. Here depen- 
dence of some special types, which occurs quite often and is of great 
importance. will be considered. We may broadly describe some stochas- 
tic processes according to the nature of dependence relationship existing 
among the members of the family. 


Processes with independent increments 
If for allif, cst. tiene SS RALIS random variables 


хи) Kt), Ха) Xt). ...› Xt) — X2 


are independent, then {X(t), t € 7) is said to be a process with in- 


dependent increments. 
Supposethat we wish to consider the 
a process in discrete time with independent increments. 


(Т tj — 1—1, X(t) = №2 


Zi = Xi-Xi-p i=l, 2,... and Zo = Xo, 
t random variables {Zn п> 0}. 


discrete parameter case. Consider 
Writing 


we have a sequence of independen 


i hastic process such that, given 
s. If X(t), t € T) is a stoc pa 
eu arb, Hf d of X(Ð, t > s, do not depend гы gr value 
X(u), u < s, then the process is said to be a Markov process. 
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A definition of such a process is given below. 
If, for, < f, <...< fact 


Pr (ac X(t) < b| XE) = xv -- Xt) = Xn) 
= Pr{a< X(1) € b | X (tn) = xy) 


the process {X(t), t € T) is a Markov process. 
A discrete parameter Markov process is known as a Markov chain. 


2.3 Stationary Processes 


If for arbitrary f, ..., tm the joint distribution of the vector random 
variables 


(X(t), X(te)s ..., Ж(һ)), апа (X(t, + 0),..., Х(һ + AY 


are the same for all h > 0, then the stochastic process {X(t), t € T) is said 
to be stationary of order n. It is strictly stationary if it is stationary of 
order n for any integer n. 

Stationarity of a process implies that the probabilistic structure of 
the process is invariant under translation of the time axis. Many pro- 
cesses encountered in practice exhibit such a characteristic. Now for 
such a strictly stationary process, the statistical properties of the asso- 
ciated joint distribution of (X(t, + А), 7. ., X(t, + /)) will obviously be 
independent of /. This will hold, in particular, in case of the moments 
of the distribution, when these exist. 

If the mean of the process {X(t)} exists, then E {X(t)} must Бе equal 
to E {X(t + h)) for any Л, so that E {X(t)} must be a constant m, indepen- 
dent of t. Without loss of generality, we shall assume that m is zero. 
If, moreover, the covariance function C(s, t) (or Сы) exists, then 


Сы = соу {X(t), X(5)) = E {Х(@) XC) 
= E{X(t +h) X(s + h}, for any h 
= E {X(t—s) X(0)). 


This shows that Cs, is a function of the time difference | t — s |. 

Higher moments, when they exist, can also be considered. Many of 
the important questions relating to a stochastic process can be adequate- 
ly answered on the basis of the knowledge of the first two moments of 
the process. Accordingly, another notion of stationarity, involving 
existence and stationarity of the moments of order one and two, is also 
considered. і 

A process X(t) with finite second order moments is called covariance 
stationary or weakly stationary or wide sense stationary (or simply sta- 
tionary), if its mean E {X(t)} = m is independent of t, and its covariance 


function 
C(s,t) = E {X(t) X(s)) 
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depends only on the difference |t—s|, for all s,s. We shall also use the 
notation C,,, for E {X(t) X(s)}, and m(t) for E {X(t)}. 

Note that a strictly stationary process will not necessarily be a weakly 
stationary process, nor will a weakly stationary process be necessarily 
strictly stationary. A process which is not stationary (in any sense) is 
said to be evolutionary. The processes considered here will be assumed 
to be.real throughout. We now define a class of stochastic processes in 
terms of an important specific distribution —normal distribution. 


Gaussian processes. If the distribution of (X(t,) ,..., X(tn)) for all f,- 
is multivariate normal, then X(t) is said to be a Gaussian process. 

Such processes derive their importance from the fact that there are 
many physical processes which are approximately normal, eg. the 
process describing atmospheric turbulence. Further, such processes 
have some very useful and simple properties. We state below such a 


property. 
If a Gaussian process X(t) is stationary, then it is strictly stationary. 
The explanation is simple. The multivariate normal distribution of 
(X), ++» ХХ) is completely determined by its mean vector (ш,,. .,ш„) 
(where p; = E(X(t)) and the variance-covariance matrix (Ci, i). hose 
elements are C(i, j) = cov (X(t), X(t;)}, i, j = 1, 2,.. „п. 


If the Gaussian process is covariance stationary, then E(X*(t)), E(X(t)) 
are finite and the covariance function C(i, j) is a function only of the 


difference i— j. 


Example 3(a) Let Y, n > 1 be uncorrelated r.v.’s with mean 0 and 
variance 1. Then 
C(n, m) = cov(X,, Xm} = E(X,X,) —0, тп 
Е =1, т=п 
and so {X,, п > 1) is covariance stationary. 


If X, are also identically distributed, then (X, is strictly stationary. 
Example 3(b). Poisson process. Consider the process {X(t),t Є T} with: 
Pr(X(t) = п} = exp (— at) (at)"/n!,a>0,n=0,1,2,.... 


We see that 

m(t) = at, var (X(t)) = at 
are functions of t. The process is evolutionary. The distribution of 
the process is functionally dependent on /. 


Example 3(c) Consider the process 


AİLE 05/4022) 
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X(t) = A, + Ast, 


where Ау, A, are independent r.v.’s with F(A;))=a, var (4) = oj 
i= 1, 2. We have 


m(t) = a, + ast. 
E(X(t) X(s)) = Е{(А, + Ast) (А, + 4,5) 
= E(A?) + (s + t) Е(А,А„\ + ts E(A,?) 
= 01 + a? + (s + {)а,а„ + ts (0 + а). 
ЕХО) = c4? + a? + 2t aya, + t? (0 + а), 
var {X(t} = cj? + to., 


Thus 


d 
v C(s, t) = cov (X(r), X(s)) = ai? + tso. 
The process is evolutionary. 


Example 3(d) Consider the process 
X(t) = A cos ot + B sin ot, 


where A, B are uncorrelated r. v.'s each with mean 0 and variance 1 and 
© is a positive constant. 
We have 


m(t) — 0, 
E{X(1)X(s)} = E((A cos ot + B sin ot) (A cos os + B sin os)) 
= COS wf COS ws + sin of sin ws 
du = Cos (s — t)o. 
E(QX()) = var (X(0)) = 1, 
C(s, t) = cov {X(t), X(s)) = cos (s — t)o. 


Here the first two moments are finite and the covariance function is ә 
function of (s — t). Thus the process is covariance stationary. 


Example 3(e). Consider the process (X(t), t © T) whose probability 
distribution, under a certain condition, is given by 


Pia) = 9} = (туж, п = 12,... 


1 
We have det 


E{X(t)} = i nPr(X(t) = п} 


п(агу—\ 
пе1(1 + аг)у"+1 
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1 E al 
=a Xoa(—25. \" 
(1+ at)? „=; (2) =l 
е 2, - 
EQ = 5 2 (at)n-i 


пе? (1 + аг)" 


E пау, n(n — 1) (; fx) ] 


> n(aty- 
APT 


=2at+ 1. 
Thus, 
var (X(1)) — 2at. 
Here the first moment is constant but the second moment (and the 
Váriance) increases with /. The process (X(1)) 15 not stationary. 


EXERCISES 
Ы! 

21. Let Y, forn even, take values +1 and — 1 each with probability 
L and for m odd, take valuesv/a,— l/v/a, with probability 
1а + 1), а/а + 1) respectively (a isa real number > — 1 and 
# 0, 1). Further let X, be independent. Show that {¥,} is cov- 
ariance stationary but not strictly stationary. (Compare this with 
Example 3 (a).) 

Let Y, =aX, + а, Xn- (n = 1, 2,...), where ag, a, are constants 
and XY, (n = 0,1,2,...) are i.i.d. random variables with mean 0 and 
variance с?, Is {У,, n > 1) covariance stationary ? Is it a Markov 


2.2. 


ргосеѕѕ ? 


23, Пели) = È (Ar cos Ot + B, sin б), 
rel 


where A, B, are uncorrelated random oe ene Р: 
variance o? and б, are constants. Show that (X(0) is c 
stationary. 

14. Let X(t) = Ay + Ait + 40, Ка ive 1; 
random variables with mean 0 and e of (X(t). 
value function and the covariance dora by 

2.5. Find the covariance function of (Y) 8! 

n-2..« 
Yp = aX, + аА + sare ; 
"sare uncorrelated random variables. 
(X0), t = 0,12. - -} with inde- 
ocess. Is the converse true ? 
cess having independent in- 


i2 0,1,2 are uncorrelated 
Find the mean 


Where a's are constants and X, 
tic process 


2.6. Show that every stochas 
. y sto 
pendent increments is à Ma АП o 
2m If (X(0, (2 0} with A(0) = 
crements, then 
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cov{X(s), X(#)} = var {X(s)}, for O<s<t 
= var {X(min (s, t))}, for any s, t (t > 0, 52 0). 


(Such a process is not covariance stationary.) 

In exercises 8-9, the distribution of №1) (the number of events 

that occur in (0, г] is assumed to be Poisson with mean df, i.e. 

{N(t), t0 is assumed to be a Poisson Process. 
2.8. Poisson increment process. The process (X(t), t > 0) defined by 

— Nt 
x = Net = NO, 

where e > 0 is a given constant, is known as Poisson increment 
process. Show that the process is covariance stationary having 


EXA} = а 
апа 
C(s, t) = 0, |t—sj>e 
42 a6t—s| |t—s| <=. 
€ e? 


2.9.(a) Semi-random telegraph signal: The process {X(t), t > 0) defined by 


X(t) = (- уч) 
is called a semi-random telegraph signal. Show that the 
evolutionary having 
E{X(t)} = exp ( — 2at) 
C(s, t) = exp {— 2a |t — ғ) — exp{— 2a ( +s). 


process is 


and 


The process {X(t)} is a two-valued process with two possible 
values 1 and —1. Itisa one-minus-one process. 
(b) Random telegraph signal : The process (Y(t), t > 0) defined by 

Y(t) =« X(t); 
where « is a random variable which assumes the values —1 and 1 
with equal probability and which is independent of X(r) (defined 
in (a)), is called a random telegraph signal. Show that the process 
is covariance stationary having 

E{Y(t)} = 0 
and 

C(s, t) = exp { —2a|t — sl}. 

(Random telegraph signals are useful in the construction of 
random signal generators ) 

2.10. Semi-binary random transmission: Assume that a fair coin is 
tossed every T units of time. The process {Z(t), t > 0} such that 
Zt) takes the values 1 or —1 according as nth toss results in a- 
head or tail respectively, where (n — 1) T < t < nT, is called semi- 
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random binary transmission. Show that the process is covariance 
stationary having 


E(Z(t)) =0 
C(s,t) = 1, if s, t are in the same tossing interval 
(n—1I1Tzst-nT 
— 0, otherwise. 
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CHAPTER 3 


MARKOV CHAINS 


3.1 Definition and Examples 


Consider a simple coin tossing experiment repeated for a number of 
times. The possible outcomes at cach trial are two : head with pro- 
bability, say, р and tail with probability q, p+ д = 1. Let us denote 
head by land tail by 0 and the random variable denoting the result of 
the nth toss by X,. Thenfor n = 1, 2, 3,..., 


Pr (X, = 1) =p, Pr (X, = 0) =q. 


Thus we have a sequence of random variables X,, Y.,.... The trials are 
independent and the result of the nth trial does not depend in any 
way on the previous trials numbered 1, 2,..., (п — 1). 


Consider now the random variable given by the partial sum 
5, = № +... + X, Thesum S, gives the accumulated number of 
heads in the first л trials and its possible values are 0,1, e «i 

We have $,,, = S, + Х,+,. Given that $, —j(Q-0,1,...0) the rv. 
$„+ Can assume only two possible values : Siu, =j with probability 
q and 5,4, =j + 1 with probability p; these probabilities are not at all 
affected by the values of the variables Situs Siege ВИЕ 


Рг {511 =J+1 | Sn =} Re. 
Pr {Sn =J | Sa = = q. 
We have here an example of a Markov* chain, a case of simple 
dependence that the outcome of (n+ 1)st trial depends directly on that 
of nth triai and only on it. The conditional probability of Sa} given S; 


depends on the value of S, and the manner in which the value of 5, 
was reached is of no consequence. 


Example Қа). Polya’s urn model : An urn contains b black and r red 
balls. А ball is drawn at random and is replaced after tbe drawing, i.e. 
drawing is with replacement. The outcome at the nth drawing is either 
a black ball or a red ball. Let the random variable Y, be defined as 

X, = 1, if nth drawing results in a black ball, and 

Xn == 0, if it results in a red ball. 


* A.A, Markov (1856-1922). 
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There are two possible outcomes of X, with 


Pr (X, = 1) 5/06 + r), Pr (X; = 0) = r/(b + r). 
We have " 
Pr (X, —J,..., X, = k} = Pr {X, =j}... РЕ{Х, = k}, J k0 


because of independence of Х,,...,Х,. 

Polya's urn model is such that after each drawing not only that the 
ball drawn is replaced but c( > 0) balls of the colour drawn (at that 
drawing) are added to the urn so that the number of balls of the colour 
drawn increases, while the number of balls of. the other colour remains 
unchanged as at the drawing, i.e. if the nth drawing results in a black 
ball, then the number of black balls for (2 + Dst drawing will bec 
more than the number of black balls at the nth drawing, while the 
number of red balls will be the same for both the drawings. We have 


Pr(X,— 1} = Pr(X,—1, X, 0 - Pri, l, X,—1) 
= Pr (X, = 1|% = 0) Pr Qr = 0) + Pr (X= 110 = 1) РЦ = 1} 


b 10 g Dub TO 
“b+r+eb+r' b+r+e bar 
= b . 
bcr 


It can be shown that Pr (Y, — 1} =b/(b + r), Pr {Xn == 0} = r/(b + r) 
forall. Again, 
Pr (X, = 1, X, = 1} 
—Pr = l, %= 1, X, -0) + Pr = 1, = 1, X, — 1) 
= Pr (X, —1|3,—1, Y, =0} Pr, = 1 | X, =0} Pr{x, = 0) 
T Pr(xX,-— ШЕ 0 X; —1)Pr(X,— l| X, 1} Pr(X, «1j 


| bcc А b Е. че b-- 2c bo. b 
b+r+2¢c тс bir b+r+2ebtr+cb+r 
а b(b + с) Д 
BODIE EE 
Hence 
ў Рг{Х„ = = 
Pr(X, = | = 1) = тот 1} 
= 
ero 
b+rt+e 
but 
Ега 
and i 
b+c 


PO = m1 X m0 LET. 
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РЕ ГААГ 
Неге the conditional probability of X, given X, is not equal to that of 
X, given X, and Xj. А А | Е 

Now, consider ihe random variable Y, which gives the number of 
black balls at the time of nth drawing. У, takes only one value b, У, 
takes values b and b + c, Y, values b, b + cand b+ 2c etc. In general, 
Y, can assume vaiues b, b + c,...,b + (n—1)c. Given that Y,., assumes 
one of the possible values b, b+ c,...,b (п — 2)с, say i = Б + kc, 
К = 0, 1,....п —2, then Y, assumes the value j = i with probability a, 
and the value j = і + c with probability b,, where 


m r 

BS Bite Oe Е. 
b 

b, = Pr ie = = Tp 


Let {X,, n = 0, 1, 2,...) be a sequence of random variables (or a 
discrete parameter stochastic process). Let the possible outcomes of 
Xn be E; (j = 0. 1, 2,...), where the number of Е; may be finite (say, т) 
or denumerable. To the event E; let us associate the value j(=0, 1, 2,.-+) 
so that the possible values of Y, constitute a set N = 10, 1, 2... and that 
the process has the state space N. Unless otherwise stated, by state 
space с? a Markov chain, we shall imply discrete state space (having 
finite or a countably infinite number of elements); it could be N or some 
other subset of the set of integers J = {жө toe д, = R S 


Definition. The stochastic process {Xa n—0,1,2,...) is called a 
Markov chain, if, for j,k, jy.. ini € N (or any subset of 1), 
Prix, = k| Xs, =f, X. = Ji ea Xo = ja} 
= Р(Х, = | =j} = pjr (зау, ` (1.1) 
whenever the first member is defined. 

The outcomes E; (or the values j) are called the states of the Markov 
chain; if X, has the outcome £; (i.e. X, =j), the process is said to be 
at state E; (or simply at state j) at nth trial. To state J (or outcome Е) 
there is no longer a fixed probability Pr{¥,, = j} but toa pair of states 
(j, k) at the two successive trials (say, nth and (л + 1)st trials) there is a 
conditional probability рд. It is the probability of transition from the 
state j at nth trial to the state k at (n 4- l)st trial. The transition pro- 
babilitics ру are basic to the study of the structure of the Markov chain. 

The transition probability may or may not be independent of n. If 
the transition probability ру is independent of л, the Markov chain is 
said to be homogeneous (or to have stationary transition probabilities). If it 
is dependent on n (in which case it is denoted by((p,;), the chain is said 
to be non-homogeneous. Here we shall confine to homogeneous chains. 
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The transition probability ру. refers to the states (j, k) at two successive 
trials (say nth and (n — l)st trial); the transition is one-step and ру is 
called one-step (or unit step) transition probability. In the more general 
case, we агг concerned with the pair of states (j, k) at two non-successive 
trials, say, state j at the nth trial and state at the (n + т) trial. The 
corresponding transition probability is then called m-step transition pro- 
bability and is denoted by pone, 


Pig) = Рг{Х„ = k | X, =). 


This will be considered in Sec. 3.2 below. 


Transition Matrix: The transition probabilities ру satisfy 
Pik > 0, Урук = 1 for all j. (1:2) 


These probabilities may be written in the matrix form 
PE ONERE (1.3) 


This is called the transition probability matrix or matrix of transition 
probabilities (t.p.m.) of the Markov chain. Pisa Stochastic or Markoy 
matrix, i.e. a square matrix with non-negative elements and unit row 
sums. 

It may be seen that the probability distribution: of Xp Xr Xen 
can be computed in terms of the transition probabilities p; and the initial 
distribution of №. Suppose, for simplicity, that г = 0, then 
Rii (Ba; Бу,» .., En Ej, Eg) 

= Pr {Xo =a, X bs Knee = hy Xam fe Man = 1 
= Pr {Xn =k | Ker Sh. Xo may PR, =). X cma) 
= Pr{xX, =k | Xn- = J} Pr Ga j| Xii) 
AP Mes d, ez c a] 
—Pri(X,—k | Х, =} Pe А И =, 
РХ =b | X, = a} Pr {X, =a} 


= (Pr (X, = а)} Pab- < -Pij рж. (1.4) 

Thus, 
Pr (X, — a, X4 = b,..., Xren-o =i, Хол =J, Xeon = К} 
= (Pr (X, = а)} pas... pij py. (1.4a) 


More generally, if we denote by Ej, the outcome of the rth trial then 


Pr (E, Ej, «++, E) = (Pr (Es) Pins Pr fae Pig ayin 
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The matrix of transition probabilities together with the ишш distribution 
completely specifies a Markov chain (Xn, a= 0, 1,2, .. J | 
We state (without proof) the general existence theorem of Markov chains. 

Given the set Л/ and the sequence of stochastic matiices (рд) == (MP 
there exists a Markov chain (X, with state space N and transition. pro- 
bability matrices, (ОР. (For proof, see Iosifescu & Tautu, Stochastic 
Processes—l, Springer-Verlag (1973), Chung (1967).) 


Order of a Markov chain 


Definition. A Markov chain (X,) is said to be of order $(5—1,2,3,...); 
if, for all л, 


Pr {Xn =k | Xi = ј, Х,, = Дв “э ONE Jesi . 3 
SEE (Mn oe MO | Мз, Sif M ACCU ECC TC 
A Markov chain {Х, 


} is said to bè of order one (or simply a Markov 
chain) if 


Pr — k Lea аа c = Prom 1X nl 

= Pik ; (1.6) 
Unless explicitly stated otherwise, we shall mean by Markov chain, a 
chain of order one, to which we shall mostly confine ourselves here. A 


chain is said to be of order zero if рук == рк for all j. This implies inde- 
pendence of Y, and P ON 


Deríote the state that a day is rainy by 1 and that a day is not rainy 
by 0, { 


Let (1.5) hold for s =2 and let 


Dijk = Pr (actual day is in state К | the preceding day was in state j, 
the day before the preceding was in state Тулу, k=O, 1, 
We have then a Markov chain of order two. 
is not a stochastic matrix. 
Let (1.5) hold for s — 1 and let 


Note that the matrix (pij) 


Pik = Pr (actual day is in state К 
We then have a Markov chain 
(рк), J; k= 0, 1, 


| preceding day was in state Ji 
(i-e. a chain of order one) with t.p.m. 


Example 1(b). Let {Xn n 
and with transition matrix 


34 14 о 
l4 02 14 (1.7) 
0 3/4 14 
and the initial distribution PE, = 


> 0} be a Markov chain with three states 0, 1, 2 


1). i0, 1, 2, 
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We have "S 
Pr(—1|]X,—-2)—-$ 
рг{Х„=2|Х=1}={ 
рг{Х„=2,Ху=1|%=2) 
= Pr {¥,=2 |X% =1}Рг{Х,=1|Җ%=2}={.@ = ře 
Pr {X, = 2, X,=1, X= 2} 
= Pr{¥, = 2, X, = Хо = 2}Рг{Х„= 2} = ү. — үн 
Prix, = 152% 2, Y]: 2] 
= Pr {% = 1 | Х„= 2, X%1= 1, X, —2).Pr (X, = 2, 
X= 1, Xj = 3} 
—Pr(X,—1]|X,—2) (s) = 4.3 


Example 1(с). А particle performs a random walk with absorbing 
barriers, say, at 0 and 4. Whenever it is at any position r (0 < r < 4), 
it moves to r + 1 with probability p or to (r — 1) with probability q, 
p+q=1. Butas soon as it reaches 0 or 4it remains there itself. Let 
Xn be the position of the particle after moves, The different states of 
X, are the different positions of the particle. (X,) is a Markov chain 
whose unit-step transition probabilities are given by 


Pr(X,—r--liXiz,—r-p 


Pr {Xn =r l| Хо =) =9 O<r<4 
Pr(X,—0]| X», =0} = 1, 
and Pr (€,—4| Xr =H — 1. 
The transition matrix is given by 
Z States of X, 
States of X,.4 0 1 2 3 4 
0 1 ОХ 40 057450 
1 VR TUR 35 o 9 
2 0 q 0 p 0 
3 0 0 q 0 Р 
4 О КО 0 0 i 


Example i(d) Suppose that a coin with probability p for a head is 
tossed indefinitely. Let Xn, the outcome of the nth trial, be k, where k 
(= 0, 1,..., п) denotes that there is a run of К successes, i.e. the length of 
the uninterrupted block of heads is k. {X,} constitutes a Markov chain, 
with unit-step transition probabilities 
Pik=Pr {Xn = k | Xn- = j} — Р, k=j+1 
=q, k=0 
= 0, otherwise. 
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The transition matrix is given by 


P States of X, 
States of Х,_, 0r I2: Sas RM LIST 
0 @ aU uu ОО: 
] ПОР ОУ - 0 TT | 
2 q^ 0 ^0! жы, d zi 
k q 0 0 0 p 


Example 1(е). Partial sum of independent random variables: Consicer a 
series of coin tossing experiments, where the outcomes of nth trial are 
denoted by 1 (for a head) and 0 (for а tail). Let Y, be the random vari- 
able denoting the outcome of nth trial and Sn=X,+...+ X, be the nth 
partial sum. The possible values of Snare0,1,...,n, ie. the states of Sn 
arer, r=0, 1, . „п. {5,) is a Markov chain with transition matrix 


P d States of Y, 
States of Xp- 1.72: r rl 
1 @ р 0 0.9 s 
2 0 qp 0219 : 
3 0 0 д 0 0 : 
r 0 00 q p 


A Markov chain {Х,, п > 0) with К States, where k is finite, is said to 
be a finite Markovchain. The transition matrix P is in this case a square 


matrix with k rows and k columns. Examples i(b) and 1(c) deal with 
finite Markov chains. 


The number of states could however be infinite. If the possible values 


Of аге. .;,—2, — 1,0, 1,2... then the Markov chain is said to be 
denumerably infinite or denumerable. Examples l(d) and l(e) are of 
denumerable Markov chains. 


3.2 Higher Transition Probabilities 


Cliapman-Kolmogoroy equation: 
one-step transition probabilities, 
the probability of the outcome at t 


We have so far considered unit-step or 
the probability of Y, given Xay Le. 
he nth step or trial given the outcome 
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at the previous step. рук gives the probability of unit-step transition from 
the state j at a trial to the state k atthe next following trial. The m- 
step transition probability is given by 

Pr {Xmen = k | Xp =j} =p; (2:1) 
Рі”) gives the probability that from the state j at nth trial, the state k 
is reached at (m + 7)th trial in m steps, i.e. the probability of transition 
from the state j to the state k in exactly әт steps. The number n does not 
occur in the r.h.s. of the relation (2.1) and the chain is homogeneous. 
The one-step transition probabilities pP аге denoted by ру for simpli- 
city. Consider 


pp — Pr (Xu, =k | Xa =j} А (2.2) 


The state k can be reached from the state j іп two steps through some 
intermediate state r. Consider a fixed value of r; we have 


Рг (Xi = k, Xu =r | X»— 
= Pr {Xna = k | Xa = r, Xn = j} Pr {Xa = r | X = 
= po po = Pir Prk. 
Since these intermediate states r for r — 1, 2,...are mutually exclusive, 
we have 
pe) = Pr { Xn =k | Xa =j} — z Pr Air khamar | Ху mj) 
IE Z Pir Prk (2.3) 


(summing over all the intermediate states). 
By induction, we have 
РИ + > Pr {Хл+т+у = k | X, = J} 
EZ Pr {Xntmer = k | Xiàm = r} Pr (Xm =r | Xn =j} 


= Х pupie. 
т 


Similarly, we get 
Р Ө = X py PRD 


In general, we have 
pinta) = ee z ppm = = z р" pm, (2.4) 


This equation is a special case of —VM— equation, which 
is satisfied by the transition probabilities of a Markov chain. 

We can putthe results in terms of transition matrices as follows. 
Let P -= (рук) denote the transition matrix of the unit-step transitions 
and P" = (Р) denote the transition matrix of the m-step transitions. 
For m = 2, we have the matrix P(®) whose elements are given by (2.3). 
It follows that the elements of P'? are the elements of the matrix 
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obtained by multiplying the matrix P by itself, i.e. 


pO = Р-Р = P?, 
Similarly, pli) — pa P pipa 
d 
an panem) — Рт. рп — pn, pm. (2.5) 
It should be noted that there exist non-Markovian chains whose transi- 
tion probabilities satisfy Chapman-Kolmogorov equation (for example, 
see Feller I, p. 423, Parzen p. 203). 


Example 2(a). Consider the Markov chain of Example 1(b). The two- 
step transition matrix is given by 


34 1/4 0 \/3/4 14 0 5/8 5/16 1/16 
4244 2 14| 516 1/2 3/16 
( 3/4 us! 3/4 us) (5: 9/16 e) 
Hence pf = Pr {Xn = 1| X, = 0) = 5/16 for n > 0. 
Thus 

Pr (X, = 1 |X, = 0} = 5/16, 
and 


Pr (X, = 1, X, —0) = Pr (X, = 1 | X, = 0} Pr (X, = 0} 
= (5/16).(1/3) = 5/48. 


Example 2(b). Two-state Markov chain. Suppose that the probability 


of adry day (state 0) following a rainy day (state 1) is 1/3 and that the 
probability of a rainy day following a dry day is 1/2. We have a two- 
state Markov chain such that p,, = 1/3 and p,, = 1/2 and t.p.m. 


0 1 
(г) 


We have 


p= 5/12 el Ра 259/432 
~ 17/18 11/8 ~ 259/648 агада], 


Given that May 1 is a dry day, the probability that May 3 is a dry day 


is 5/12, and that May 5 is a dry day is 173/432. For calculation of 
powers of P, refer to Sec. 3.4. 


/ 


Example 2(с). Consider a communication system which transmits the 
two digits 0 and 1 through several stages. Let {Xn n > 1} be the digit 
leaving the nth stage of system and X, be the digit entering the first 
stage (leaving the Oth stage). At each Stage there is a constant proba- 
bility q that the digit which enters will be transmitted unchanged (i.e. 
the digit will remain unchanged when it leaves), and probability p other- 
wise (i.e. the digit changes when it leaves), p +q = 1. 
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Here (X,, n > 0} is a homogeneous two-state Markov chain with 
unit-step transition matrix 


"o 


It can be shown (by mathematical induction or otherwise) that 
pn -(i FIU -D T i-—314— m | 
4—aq—p"  4+4(4—Р)" 
Неге 
ре = р = (9 — Р)" 
and 
ра = рзд =4—4(9 — Р)". 
and as т-> со, 
lim p% = lim p = lim ру = lim pS? — 3. 
Suppose that the initial distribution is given by 
Pr {X, =0} =a and Pr {X= 1) =b = 1 – a. 
Then we have 
Pr {Xm —0, X, —0) = Pr (X, = 0| X, = 0) Pr (X, = 0) 


= ар“? 


апа 
Pr {Xm = 0, X, = 1) = bp. 


The probability that the digit entering the first stage is 0 given that the 
digit leaving the mth stage is 0 can be evaluated by applying Bayes’ rule. 
We have 
Pr (X, =0| Xm = 0) 
als Pr (X, = 0 | X, = 0) Pr (X, = 0) 
= Бр = 0| X, = 0j Pr (X, = 0) + Pr {Xm = 0X, = 1) Pr{Xo = 1} 


) 
_ aps 


= a{g3+3(g—p)"} 

аф +4 (q—p)"} + bi — #(4—Р)"} 
aal taa, 

1+ (a — b) (a — p)" 


3.3 Classification of States and Chains 


The states j, j = 0,1,2,... of a Markov chain (X,, n > 0) can often be 
classified in a distinctive manner according to some fundamental pro- 
perties of tlie system. By means of such classification it is possible to 
identify certain types of chains. 
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If the probability ng is non-zero for some n > 1, then we say that the 
state k can be reached from the state j. If every state can be reached 
from every other state (in any number of transitions) the chain is said to 
be irreducible. The transition matrix is then irreducible. 

If C is a set of states such that no state outside C can be reached 
from any state in C, then thesct C is said to be closed. If C is a closed 
set and j € C while k ё C, then pj; = 0. It can be seen that p 0, 
since one of the factors in each term of the sum given by (2.3) will be 
zero. More generally, ру = 0 forn > 1. 

A closed set may contain one or more states. Ifa closed set C con- 
tains only one state j, then the state is called an absorbing state. jis 
absorbing iff p;; = 1 and then ру — 0 for k 5 j. The set of all states 
of a Markov chain will be a closed set. Itfollows that if a Markov 
chain does not contain any other closed set (with the exception of the 
set of all states) then the chain will be irreducible, i.e. an irreducible. 
chain does not contain any proper closed subset. Chains which are not 
irreducible are said to be non-irreducible or reducible, or decomposable. 
Every Markov chain will contain at least one closed set, i.e. the set of 
ап states. If the number of closed sets is two or more, the chain is 
reducible. 

In Example 1(с), the states 0 and 4 are absorbing states. In Exam- 
ple 2(c), forp 5 0, 1 (and д = 0, 1) both the states can be reached from 
either of the states, i.e. the Markov chain is irreducible. 

We now proceed to obtain a classification of the states of a Markov 
chain. 

Suppose that a system starts with the state j. Let/Í?be the proba- 
bility that it reaches the state k for the first time at the nth step (or 
after n transitions) and let pf? be the probability that it reaches the state 
К (not necessarily for the first time) after л transitions. A relation can 
be'established between f$ and p 2 as follows : 

тие probability that starting with j, the state k is reached for the first 
time at the rth step and again after that at the (n — r)th step is given by 


IR pu? for allr < n. These cases are mutually exclusive. Hence 


n 
PR= Y Pk nl (3.1) 
3 
with 
Р = 1, fR = 0; 790 = Px 
or, 
n—l 
P= Х 530 ри? +R п> 1. (3.2) 


Let Fik denote the probability that starting with the state j the system 
will ever reach the state k. Clearly 


Fy= Xf. (3.3) 
nel 


MARKOV CHAINS f 65 
The mean (first-passage) time from the state j to the state k is given by 
шк = A nf De (3.4) 

n= 


We have to consider two cases, Fj, = 1 апа Fy, < 1. 

When Fj, = 1, it is certain that the system starting with the state j will 
reach the state k; in this case (/ 2, n = 1, 2,...} is a proper probability 
distribution and this gives the first passage distribution for k (the 
system starting with j). In particular, when k =j. {/0,п = 1, 2...) will 
represent the distribution of the recurrence times of j; and Ру = 1 
will imply that the return to the state j is certain. In this case 


uj X mfp (3.5) 


n=1 
is known as the mean recurrence time for the state j. 


Thus, two questions concerning state j arise: first, whether the return 
to the state jis certain and secondly, when this happens, whether the 


mean recurrence time u;; is finite. 


Definitions. The state jis said to be persistent* if Fj; = 1 (i.e. return to 
the state j is certain) and transient if Еу; < 1 (ie. return to the state j is 
uncertain) А persistent state j is said to be null, if uj; = оо, ie. if the 
mean recurrence time is infinite, and is said to be non-null if u;; < oo. 
A state j is said to be periodic with period (> 1) if return to the state is 
possible only at t, 2t, 3/,... steps, where / is the greatest integer with this 
property. In this case pj?! = 0 unless m is an integral multiple of f. In 
other words, if ¢ is the greatest common divisor of all integers n( 2 1) 
for which p? > 0, then j is said to be periodic with period t. The state 
j is said to be aperiodic (or non-periodic) if no such t( > 1) exists, e.g., 
1 = 1 will correspond to the aperiodic case. А persistent, non-null 
and aperiodic state (of a Markov chain) is said to be ergodic. A 
Markov chain all of whose states are ergodic is said to be an ergodic chain. 
We consider below some important results concerning classification of 


states. 


Theorem 3.1. The state j is persistent or transient according as 


z pf? = co or < vo. (3.6) 
n-0 


Proof: Let 


Z pp 5", |si<l 


nel 


Pj; (5) = 5 рї) 5" =1 + 
neo ^ 


*The word recurrent is also used by some authors; we shall however use the word 
persistent. 


5(45-95/1982) 
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and 
y fH бп (n) gn. 
Fj;(s) = РА Jost ESI 5", |s| < 1 


be the generating functions of the sequences (po) and {70} respectively. 
We have from (3.1) 


n 
= r) p(n—r. =f 
pp Erg aon. QU 


Multiplying both sides of (3.7) by. s" and adding for all n > 1 we get 
Ру (5) - 1 = Fi; (s) Py; (5). 


The r.h.s. of the above is immediately obtained by considering the fact 
that the r.h.s. of (3.7) is a convolution of {f;;} and {ру} and that the 
generating function of the convolution is the product of the two generat- 
ing functions. Thus we have 


1 
E 28743 38 
Pj(s) = ay 151 1 (3 8) 
Using Abel’s lemma, we have as s > 1, 
1 
Pj; (s) > ГЕ; when Fy; < 1 
— co, when £j; = 1, (3.9) 


which proves the theorem. Д 


Corollary: If j is transient then pf? — 0 as n — со. 
This follows at once from the fact that when state is transient, the 


series Z pf) converges. 
5 


We now state a lemma without proof (for proof of this important 
basic limit theorem of renewal theory, see Prabhu, Feller). 


Lemma 3.1 Let {f,} be а sequence such that f, = 0, f > 0, Ху, = 1 
апа t(> 1) be the greatest common divisor of those п for which f, > 0. 
Let {un} be another sequence such that uo = 1 and Un = ип (п > 1). 

$ rl 


Then 
lim us = t/p, (3.10) 
п> = 


where p= X n f, the limit being zero when u = со; 
п=1 


and lim uy = 0, 
№» 


whenever N is not divisible byt 


Theorem 3.2. If state j is persistent, then asn — oo, 


(i) p S) tly (3.11) 
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when state j is periodic with period г: 
Gi) ; Р! > Vig (3.12). 
when state j is aperiodic. 

In case state j is persistent null (whether periodic or aperiodic), 


p90,  asn- о. (3.13) 


Proof: Let state j be persistent; then p; = ® п/у is defined. 
Since (3.7) holds, we may put В 

SEP for fma р! for tn, and руз for v 
in the lemma 3.1 above. Applying the lemma, we get 


PGP > taj, as n oo, 


when state j is periodic with period f. 
When state j.is aperiodic (i.e. t = 1), then 
PG > 1ш as n co. 
In case state j is persistent null, пуу = ©, 
and 
po 0 asn> о. A 
Note that a persistent state j is non-null or null according as lim p $? is 
positive or zero. i 


Theorem 3.3. If state k is either transient or persistent null, then for 
every j 
pi --o asn-» co (3.14) 


and if state k is aperiodic persistent non-null, then 


рі? Fit. аѕ п > o. (3.15) 
Hkk 


Proof: We have 
n 
=È SR n 
Let n >m, then 
" m n 
P= Зур reer + i row 
r= г=т+1 


<Ў/@ю ? + X fy (3.16) 


r=1 r=m4+1 


Since state k is either transient or persistent null, phir? > 0 as п co 
(Theorem 3.2). 


E 2 
Further, X 790 < оо and Z fP>O0asn,m> c. 
m=1 т=т+1 
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Hence as n — co 


Pp => 0: 
From (3.16) 
n n 
Pie ZIRE < 2 py (3.16a) 


Since j is aperiodic, persistent and non-null, then by Theorem 3,2, 
Dx = Mug as n co. 


Hence from (3.162) we get, as n, m — co 


(п) tn Tie 
ТИБ зава 
Pik бшу, 
Ек 
(m ^. 
or IEF ae СА 


Theorem 3.4. In an irreducible chain, all the states are of the same type. 
They are either all transient, all persistent null, or all persistent non-null. 
All the states are aperiodic or periodic and in the latter case they all 
have the same period. 


Proof: Since the chain is irreducible, every state can be reached from 
every other state. If i, jare апу two states, then i can be reached from 
j and j from i, i.e. 


p) =а>о for some N21 


and р? —bz0 for some М> 1. 


We have р" — рот) урт) ptm 
Е 


> PP p for each r. 


Hence 
ри > рО) ppg =ab р\ў (3.17) 
апа DANEM) > pe р {P pt? = ab p». (3.18) 


jis 


From the above it is clear that the two series 5 p(?, and = pf? converge 
n n 

or diverge together. Thus the two states ijare either both transient or 

both persistent, 


Suppose that i is persistent null, then pi? > 0 as n— co ; then from (3.17), 


p) > Oasn—> œ, so that j is also persistent null, i.e. they are both 
persistent null. 


Suppose that i is persistent (non-null) and has period t, then p(? > 0 
whenever n isa multiple of £. Now 


(NEM) 


M, 
рй (N) (M) 


2 Pü pj =ab> 0, 
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so that (X + M)is a multiple of t. From (3.18) 


р" > рр > 0 


Thus (л + N + M) is a multiple of тапа so ris the period of the stave 
Jalso. А 

Two states are said to belong to the same class iff each one of them 
can be reached from the other. A property defined for all states of a 
chain is called a class property iff its possession by one state in a class 
implies its possession by all states of the same class. With these defini- 
tions, the above theorem can also be stated as follows : 

In an irreducible chain all states are of the same class. The property 
of being recurrent (transient, persistent null, persistent non-null) is a class 
property. So also is the property of having a period. 


Example 3(a). Consider the Markov chain with t.p.m. 


0 1 2 
0 0 1 0 
P=1 + 0 + 
2 0 1 0 


i 0 4 
^-( 2 1 о), pip: 
i 0 + 


in general, 
2n — p2, pint] — р, 


so that 
pe > 0, pf"*?—0 for each i. 


The states are periodic with period 2. It follows that Epí? diverges, so 
n 


that the state i is persistent. 
(Note that a periodic state must necessarily be persistent, though a 
persistent state may or may not be periodic.) 
We find that fi, =0,/{2 =1 so that Fy  Zff? = 1, ie. state 1 is 
n 
persistent and hence the other states 0 and 2 are also persistent. 
Now © 
py, = af =1x0+2x1=2, 
п 


Thus the states of the chain are periodic (each 


ie. state 1 is non-null. 
Further, 


` With period 2) and persistent non-null. 
nu эы = 2/2= 1 for all n. 
3.4 Determination of Higher Transition Probabilities 


Consider a Markov chain with т states (m finite) baving transition 
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probability matrix (t.p.m.) P = (pij. The n-step transition probabilities 
PSP (i.e. the elements of P^) can be obtained by using 

рі? = Хра”, mE, 2. е 


where pPP=1, a= kjp = pr 


We shall now show howsome of the results of spectral theory of matrices 
could be used to determine 209) ог Р" more expeditiously. We recall the 
discussions on matrices given in Sec. 1.7, Chapter 1. 

Assume that the eigenvalues /;, і = 1, 2,..., m, of the stochastic 
matrix P are all distinct and different from zero. Suppose that 

X= (Хи, Xin- +5 Xim) 

and Yi = (p Yiz- -s Yim) 
are, respectively, the right and left-eigenvectors of P corresponding to f, 
and that X = (X,..., Xm)is the matrix formed with the eigenvectors 
Xy...» Хм. Suppose that D = (dj) is a diagonal matrix such that 


di = 1, di; з= 0,/ zi. 
Then from the spectral theorem we have 
PÉXDXA (4.1) 
and p^ — XD^x-1 (4.2) 


It fk" Ck Хк yk , Where ск = 1/(ук' хк). 
m 
Thus ` py = tk” Ck Xi ук» N= 0, 1, 2,... (4.3) 


Aperiodic Chain: Limiting Behaviour 

Assume that P is primitive, ie. the chain is aperiodic; then there 
will be no other cigenvalue with modulus equal to 1 except 1. Since all 
the eigenvalues are assumed to be distinct, we may put 


fes To ш |=& 1, * 
Then хи = 1 for all i (Theorem.1.13) and we have 


m 


BO = сууу; Pe I? Ck Хы Ук (4.4) 
Further, in the limit, as п > oo, 
m 
PP > суу» (у= Ms x) = VZ xu) (4.5) 


i.e. PU? tends to a matrix with all rows equal to 


" Ax "Yt J 
Сууу = (an Cam + +s т). 
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The above shows that lim рї) or Р) exists and is independent of the 


initial state ї, if P is primitive (sce also Theorem 1.14). 
To calculate this limit, when it exists, one needs determine only the 


left-eigenvectors y,’ corresponding to /, = 1. 


Remark: The above method used in the study of limiting behaviour of 
apertodic chains is based on spectral theory of matrices. A simpler 
method for deriving lim pf? will be discussed in the next section. 


Example 4(a). Consider the two-state Markov chain 


1—4 а 
= b 
P ( b 126.02 Epi 
The eigenvalues of P are г, = 1 and f, = 1 — a — b,| t, | x 1. The right 
eigenvectors corr esponding to f І and /, = i a — b are given, res- 
pectively, by x, and x,, where x, = (1, 1) and хз = (1, — a/b). The left 


eigenvectors corresponding to г, and f, are given by y,' = (1, a/b) and 
y, = (l, — 1) respectively. We have 


c, = M/(yixy) = b/(a + Б), с, = 1/(узх,) = аа + b), 


"mw азу xy -( 1 -- 1 
X = (i ajb) “3: ~\—bja — bla] 
Thus 


а иа ba Шаруа а 
Pad пеку? = аъ (5 3 a+b E в). 


1 (b a 
ет 


+ 
OLET 0 @ п=0,1,2,.... 


a+b br vb 
Further, 
cox (Бау рка) aate бй) 
ae B= a+b (> 4 (ota 4 b) а|(а+ 5i 


As n> со lim pr, > b/(a + b) and lim pj, > a/(a + b), iR1,2.* (4.6) 
Example 4(b). Consider the three-state Markov chain with t.p.m. 
0.5 0.3 0.2 
Р= 10.2 0.4 0.4 
0.1 0.5 0.4 


The eigenvalues of P are t, = 1, t, = 0.1, t, — 0.2. The left eigenvector 
Y = (Yur Уш» Yrs) corresponding to t, = 1 satisfies 
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—0.5y, + 0.253. + 0.154, — 0 
0.3y,; — 0-642 + 0.513 = 0 
0.2у + 0.4) 12 — 0.64, = 0 


We have y, = (0.16, 0.28, 0.24) and c, = I/( i) — 1/0.68. Hence 
j- 


D > 05 (J= 1, 2, 3), 


ie. pe) — 0.2353, pi) — 0.4118, р) — 0.3529. 


Example 4(c). For the three-state Markov chain with t.p.m. 


0 1 0 
P=\0 0 1]. 
1 0 0 


The eigenvalues аге /, — l, t, = о, f; — w, o, 0° being the imaginary 
cube roots of unity. Аѕ | о | — [0° | = 1, the matrix is not primitive; 
the chain is periodic with period 3, (P? — /). Equation (4.5) cannot be 
applied in this case. 


Let us now examine some of the restrictions imposed in deriving 
(4.1)-(4.5). 


(a) The results hold even when the chain is denumerable. 

(b) Suppose that 0 is an eigenvalue (say fm == 0) and the other eigen- 
values tj, i= 1,..., m — 1 are distinct and non-zero. Then (4.2) 
holds for n = 1, 2,... with t, = 0, and so (4.5) also holds. 

(c) In the discussion of the spectral representation of a matrix (Sec. 
1.7C) it is assumed that the eigenvalues of the matrix are all dis- 
tiact. Suppose aow that the characteristic equation has some 
multiple roots. In this case there will be some modification in 
the spectral representations of P and P" given in (4.1) and (4.2). 


Before the spectral representation in the Бепега! case (with multiple 

root) is given, the following categories of multiple roots may be noted : 

G) 4 = 115 a multiple root irrespective of whether some of the other 

roots 1, i 1 are multiple. This case which is of special impor- 

tance occurs whenever the chain contains two or more closed 
subchains. 

(ii) t, = 1 is not a multiple root but some f;(i # 1) are multiple roots. 


Consider the general case when 7, i= 1,. .., k are multiple roots of order 
+ к 
r( > 1) respectively, i.e. 2 ғ; = m, the order of the chain. In this case 
1=1 


there exists а non-singular matrix H such that Р has the canonical repre- 
sentation, Р 
Р = HZH, (4.7) 
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where 
Е (ty) 0 
Ze Ar, (te) 
2 (4.7a) 
0 А1) 
and A, (t) = (ai,; (t)) is a (r x r) matrix such that 


а„ (t) = t when i =j, ai; (t)=1, when j=i +1 
and “anj (t) =O for j # i, i+1. Clearly, A, (0) = a, — t. 


Now it can be seen that 


Г" [2 ss nr (51) п-га 7] 


oy - | o mw le Л 2) pore |, (4.8) 
LIE eo J 
where (С ) = 0 for n < К. 
We also get 2” by replacing 4,(t) in the r.h.s of (4.7) by (4,(0)). Thus 
P" = Н Z" H~, n=0, 1, 2, +... (4.9) 


In the general case with multiple roots we thus have (4.7) and (4.9) in’ 
place of (4.1) and (4.2) respectively. 


3.5 Stability of a Markov System: Limiting Behaviour 
Definition. Stationary distribution: Consider a Markov chain with 


transition probabilities рж. A probability distribution {уу} is called 
stationary (or invariant) for the given chain if 


у = 2 Yj Pik 
such that y; 2 0, 4 уу = 1. 
Again Y je Vj Dijk = 7 { 2 ир} Dik => "np 
and, in general, Yk =z vpih n 2l. 


We now study the question—whether a Markov system, regardless of 
reaches a steady or stable state after a large number of 
In other words, under what conditions, if any, as 
When such limits 


the initial state /, 
transitions or moves. 
n tends to оо, pi) tends to a limit v; independent of j. 
exist, the system settles down and becomes stable. It then shows some 
long run regularity properties. 

Theorem 3.6 gives a sufficient condition for the existence of ук. It also 
shows how the distribution {у} can be obtained, in case where this 


exists. 
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We first prove Theorem 3.5 which will be needed to prove Theorem 3.6. 


Theorem 3.5. If the state j is persistent, then for every state k that can . 
be reached from state j, Fix = Ж 


Proof: Let ак be the probability that starting from statej, the state 
k is reached without previously returning to state j. The probability of 
never returning to state j once state k is reached is (1— Р). The pro- 
bability of the compound event that starting from state j, the system 
reaches state k (without returning to state j) and never returns to state j 
is ак (1 — Ем). If there are some other states, say r, 5, ..., then we get 
similar terms a, (1 — Fj), as (1 — Fis), .... Thus the probability that 
starting from state j, the system never returns to state j is | 


Р = ак (1 — Ек) + а, (1 — Fie) + ar (1 — Fj) +... ý 
But since state j is persistent, Fj; = 1 and the probability of never return- 


ing to state jis 1 — Fj;=0. Tbus P=0. This implies that each term 
is zero, sothatF,,=1. А 


Theorem 3.6. For an irreducible ergodic chain, the limits 
ye = li (л) 
ук = lim р (5.1) 


exist and are independent of the initial state j. Тһе, limits ук are such 
tha vy > 0, Z v = 1, i.e. the limits v; define a probability distribution. 
Furthermore, the limiting probability distribution (vy) is identical with 
the stationary distribution for the given chain, so that 


vk d s Vj Pik ®ук=1. (5.2) 


Writing V = (у,,...‚ ук,...), Хук = 1, the relation (5.2) may also be 
written as 
V=VP 


or V(P — I) = 0. " Ga 


Proof: Since the states are ergodic, i.e. aperiodic, persistent non-null, 
lim PR exists and is equal to Fjx/,,, (Theorem 3.2). Again, by Theorem 
п>= > 
3.5, we find that Fj, = 1, so that 
ук = lim р? = 1/ш > 0, 
ne ЭЁ 


and is independent of j. 
We have 
purto = > pp? pe» 


yy == lim pt?) = lim Z рб) pt» 
no Jk nę% і Pit Pik 


> 2 (нш Р?) p, (by Fatou’s lemma’ 
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= ly Р, for all m, 
i 


Le. Ук > È vi por. 
i 
Suppose, if possible, that ук > Z v; р; then summing over all k, we get 
i 
Ly > 5и ри) = У, 
k ki i 
which is impossible. Hence the sign of equality holds, i.e. 
ук = X vi pi 
i 
for all m > 1. In particular, when т = 1 
Ук = T Yi Pik. 


For large in, we have 
ук = È уук = (È vi) Ук. 
i i 


Hence Z v; = 1. Besidesshowing that (v) is а probability distribution, 
the normalising condition (Ху; = 1) ensures that the distribution is 
unique. A , 

We state, withort proof, the converse which also holds. 

If a chain is irreducible and aperiodic and if there exists a unique 
stationary distribution {vz} for the chain, then the chain is ergodic and 
ук = luas. 

Thus ergodicity is a necessary and sufficient condition for the existence 
of {vx} satisfying (5.2) in case of an irreducible and aperiodic chain. 

Note : The matrix (P—J) in (5.3) is such that its off-diagonal elements are non- 
negative and the diagonal elements are strictly negative. From these properties and 
the irreducibility condition it follows that the matrix (P—J) is of rank (m—1), where 
m is the number of states of the chain. If the number of states is finite, then {ук} can 


be obtained from the homogeneous equations V (P—71)- 0 (which are finite in num- 
ber), and the normalising condition Ev, —1. Some examples are considered below. 


Example 5(а). Consider the two-state Markov chain of Example 4(a). 
The chain is irreducible and consists of two persistent non-null states. 
The equations (5.2) giving v}, v» become 

ау — bv, =0 and v, + у, = 1, 
whence we get v, = b/(a +b) and у, = а/(а +b). These were directly 
Obtained as limits in Example 4(a). 


Example 5(b) Consider the three-state Markov chain having t.p.m. 
0 2/3 1/3 
1/2 0 1/2 
1/2 1/2 0 
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The chain is irreducible and the states are persistent non-null. The 
equations (5.2) giving Ук (k =1, 2, 3) become 

0=— v+ $v + in 

0—2»—»,d i» 

] —»- TY 


t 
WEER 9, = 13, Vp = 10/27, », = 8/27. 


Example 5(c). One-dimensional random walk with two reflecting bar- 
riers. Let the states be 1,2, ..., k. The particle moves from state i to state 
(i +1) with probability p and to state (/— 1) with probability q = 1 — p 
for i=2,...,k—1. From states 1 and k the particle is reflected, i.e. 
from state 1 it moves to state 2 with probability p and remains at state 
1 with probability q and similarly from state k it moves to state (k— 1) 
with probability g and remains at state with probability p. 
The transition probabilities are : 


Д3 J Stl 
=q, j=i-1 for 1<і< Е 
pi=0, izlk 
Piz FP, Pin = 9 
Pkk = р, Pkk- == 9. 


The chain is irreducible and the states are persistent non-null. The 
limiting probabilities ух are solutions of the equations (5.2). The 
solution can be obtained recursively. From the first (k — 1) equations 
we have 


Vo = (pÍq)vi , Уз== (р/9)° vy ..., Ve = (pq) v,. 
Using Ууу = 1, we get 

уу = (1 — Cla) — (ola). 
Hence 

v, = (1 — (р/4)) Cola)? (e/g), ғ= 1,..., k. 


So far we have considered finite chains. If a chain is denumerable (or 
infinite) the number of equations given by (5.2) (j, k —0, 1, 2, ...) will 
be infinite involving an infinite number of unknowns. Then the question 
of solving these equations may not arise; however, by using (5.2) 
it is possible to derive the limiting distribution in terms of its generating 
function. The following example will illustrate this point. 


Example 5(d) Consider a chain with an infinite number of states 
(0, 1, 2, ...) having transition probability matrix 
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Ро Py P» Pa 
Po Pr Р» Рх 
0 Ро n P» , 
0 0 Po Py 
where Z px — 1. 


Let 
P(s) = Z p, s* and V(s) = È vy s* 
k k 


be the generating functions of {р} and (vj) respectively. The chain is 
irreducible and aperiodic. The states are all transient, persistent null or 
persistent non-null according as P'(1) > 1, = 1 or < 1 (see Prabhu p.63). 
Thus when P'(1) < 1, vs are the unique solutions of the equations (5.2). 


The equations (5.2) become : 
Vo = Po Vo `F Poi 
уу == Py Vo + Pii + Ру!» (5.4) 
Vo == [a Уу + ру + Pie + Роуз 


Multiplying both sides of ће (k + 1)st equation of (5.4) by s* (k=0, 1, 
2, ...) and adding over k, we get 
V(s) = vyP(s) + v P(s) + vssP(s) + vs?P(s) +... 
RS) (o + (Ms) — vols) 


This gives 
V(s) == w(1 — 5) P(s)/(P(s) — 5), 
in terms of v, which can be evaluated from Z у; -= 1. We have 
lim Vs) — іт {1—5} (в). 
sol Vy zb SECS) ss 
whence 
y = 1— Р'(1) (20. 
Thus MS 
UP Gps 
a ~ P(s)— $ i 
Graph theoretic approach | 
A simple. graph theoretic formula for finding v, has recently been 


The formula, which is similar in spirit 
las, is conceptually simple and is 
Further, the formula is 
This 


advanced by Solberg (1975). 
to the well-known flow-graph formu 
claimed to have computational advantages. 
applicable to the continuous parameter case also (see Sec. 4:5D). 
approach is considered below. К 

A directed graph may be used as 2 transition ‹ 
chain. In other words, the states and the transitions betw 


ion diagram of a Markov 
een states of a 
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Markov chain may be represented by a directed graph. Here the states 
of the chain correspond to the points of the graph, the transitions corres- 
pond to the arcs and the transition probabilities to the arc weights. 


Definition. An intree T; to a specific point jin a directed graph G, is a 
spanning subgraph of Gin which every point but j has exactly one arc 
emanating from it and j has none. Thus an intree to point j is just a 
tree with the arcs all oriented towards the point j. 


The weight w(T;) of an intree jis defined to be the product of the 
weights of all arcs appearing in Tj. 


We state the theorem due to Solberg (for proof refer to his paper). 


Theorem 3.7. The steady-state solution of an irreducible Markov chain 


(discrete or continuous parameter) with a finite number of states is given 
by vx, 


ук = си Cj. 
j 


where c; = X о (Гу) is the sum over allintrees to the point j in the transi- 
i 
tion diagram of the process. 


The above theorem is illustrated with the help of an example. 
Consider the Markov chain of Example 5(b). 
The intrees to the point 1 are shown below: 


v2 1/2 


Fig. 3.1 


The weights of the intrees to the point 1 are: 


(Ту) = ры Par = 1/4, WoT) = Pos Par = 1/4, WAT3) = рур = 1/4 
so that i 


Су = ХиТ) = 3/4. 
The weights of the intrees to the point 2 are: 


Wi(75) = Pos ру, = 1/3, WaT) = р, ру = 1/3, WoT) = Pia Pe = 1/6 
so that 
t= Zw(1) = 5/6. 


The weights of the intrees to the point 3 are: 
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4(T) = PysPo3 = 1/6, Wo(T3) = Por Pis = 1/6, (73) = Pye Pog = 1/3 


so that 
Cg = 2 w((T,) = 2/3. 
1 


Thus ; 
Ec; = 3/4 + 5/6 + 2/3 = 9/4. 
We have 
» — eZ c; = 1/3 
у == cy 2 c; = 10/27 
and yy == eZ Су = 8/27. 


3.6 Statistical Inference for Markov Chains 


Inference problems, such as estimation and hypothesis testing, involving 
Markov chains have been considered by several authors (e.g. Bartlett, 
Whittle, Anderson and Goodman, Billingsley, and so on) not only because 
of their theoretical interést but also for their applications in diverse areas. 
Methods put forward for estimation of transition probabilities under diffe- 
rent situations include a recent one involving linear and quadratic pro- 
gramming procedures to produce least squares estimates (Lee ef al., 1970). 
We shall discuss here the maximum-likelihood method of estimation of 
transition probabilities from individual or micro-unit data. Some tests 
based on these estimates will also be discussed. 


Maximum-likelihood estimation: Consider a time-homogeneous Markov 
chain with a finite number, т, of states (1, 2,..., т) and having transition 
probability matrix Р = (p;x), j, k= 1, 2,..., m. Suppose that the 
number of observed direct transitions fromthe state j to the state k is 
пук, and that the total number of observations is (N + 1). Put 


m m 
2, UES and P = Bok, 95: ug Bt 


That there is a striking similarity between a sample from a Markov chain 
and one from a set of independent multinomial trials has been observed, 
among others, by Whittle (1955) who has obtained the exact probability 
of the observed лук in the form: 
' П (я;.)! 
т) == ik: 
Sow = п) үтү СТО nm pi; (6.1) 
j k 

the factor Т(лук) which involves the joint distribution of the n;.'s is 
independent of the р. 


80 STOCHASTIC PROCESSES 
The logarithm of the likelihood function can be put as 
тот 
LC pip) = C + 5 зе log pix,” (6.2) 


where C contains all terms independent of ру. 


Since È pj, — 1, (6.2) can'be written as 
k 


t m 


m t 1 m m— 
L(pi)-— C+ X Z ny log pir + X nym log (1 = Ж ps) (6.3) 
j=l k=1 j=l k=l 


Let r be a specified value of j. The maximum-likelihood estimates Pik are 
given by the solutions of the equations 


IL pr) 
=0, =h Be =); 
Әрк k 2,..., (n — 1) 
These equations give 
TE wt —0, k-12,...,(m— 1). (6.4) 
Prk 
1— Z pu 
kel 


To fix our ideas, let us take a Specified value, s, of k. Then 


peo Me oce deco o arena ONS SANs 
Prs Prk i Ei a 1 5, (m — 1) 
1 
Thus 
m~l Nem 
1— > ра= =" p, (6.5) 
kel Mrs 
and Pm E ры, Ё == 1,2, inss (m — D. (6.6) 
Summing (6.6) over all k and adding (6.5), we get 
Z ne 
k=1 


and hence the estimate Drs is given by 


^ Tr. 

P=- (6.7) 
У nk 
kel 


Now г, s are two arbitrary values of j, К respectively. 
Jj, k—1,2,..., (m—1), 


Hence, for 


^ пк Nik 
Pik =; = 1 


(6.8) 
У nk 
kei 


nj. 


are the maximum-likelihood estimates of Dix. 


| 
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Hypothesis testing: Some of the tests developed (Anderson and 
Goodman) on the above estimates are given below. 

(i) Suppose that one wishes to test the null hypothesis that the 
observed realisation comes from a Markov chain with a given transition, 
i.e. matrix P^, suppose that the null hypothesis is 


Ap РР", 
Then, for large N, and for Dix given by (6.8), the statistic 
gu. (Pik — = 
к=1 Ph. 


is distributed as X? with m — l d.f. (degrees of freedom). Here р?" 
which are equal to zero are excluded and the d.f. is reduced by the 


xd Те ets 


number of ps equal to O. Alternatively, a test for all Dik can be 
obtained by adding over all j, and the statistic 


m m ne (dp — ps)? 
z X ту. (Pn — Pin) (6.9) 
je gut jk 


has an asymptotic y?-distribution with m(n — 1) d.f. (the number of d.f. 
being reduced by the number of pj/'s equal to zero, if any, for j, k —1, 
254931): 

The likelihood ratio criterion for А, is given by 


Thus, under the null hypothesis, the statistic 


—21о80=22®я1ов („у Ур (6.10) 


has an asymptotic X*-distribution with m(m — 1) d.f. 

(i) The maximum-likelihood estimates could also be used to test the 
order of a Markov chain. For testing the null hypothesis that the chain 
is of order zero, і.е. Hy:pjx = px for all j, against the alternative that the 
chain is of order 1, the test criterion is 


^ ^ 
where pk n.k( È È пу), рук = njn. 
Pk 


Under the null hypothesis, the statistic 
nj; (п) 


— 2logA -: 222 nil 
s j k rc (n .a)Qj-) 


6(45-95/1982?) 
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has an asymptotic 7?-distribution with (т ЕА Similar test can be 
constructed for testing the null hypothesis that the chain is of order one 
against the alternative that it is of order two. 

These tests do not prove that the observed chain is of order one. 
They provide, however, a rationale for using a Markov model. Similar 
likelihood ratio tests based. on maximum-likelihood estimates are given 
(iii) for testing the stationarity (or time-homogeneity) of a Markov chain, 


ie. Hy : pj(!) => рук and also (iv) for testing that several samples are from 
the same Markov chain of a given order. 


Determination of the order of a Markov chain by MAICE 


A. procedure for the determination of the order of a Markov chain by 
Akaike's Information criterion (AIC) has been developed by Tong 
(1975). The AIC is defined as 


AIC — (— 2) log (Maximum-likelihood) + 2 (Number of independent 
parameters in the model) 


measure of deviation of the fitting model 
from the true structure. Given several models, the procedure envisages 
adoption of the model that minimises the ATC and is called Minimum 
AIC Estimation (MAICE). (It is argued that the MAICE procedure 
Tepresents an attempt to strike a balance between overfitting which needs 
more parameters, and underfitting which incurs an increased residual 
variance.) 

Denote the transition probability for a r chain by Dis. 1600,25. < 8s 
s being the (finite) number of states of the chain. 


This statistic is introduced asa 


Denote the ML estimates of p by ру. ШЫ 


D aet em crei 
Ti]...k. 
where Ajek =È Nikt. The hypothesis tested is Ну-у раја рр. 
1 


i— L.. , s (that the chain is (r — 1)-dependent against 77, : that the chain 
is r-dependent). The statistic constructed is 


Quia 5 (0ч гем) 


mr = —– 2 С: er oe co eet 
14, 2 log №; д (т... м) log tamer 


which is à X*-variate with sls — 1)? d.f. 
The hypothesis Hz that the cham is k-dependent implies the hypo- 
thesis Н, that the chain is r-dependent fork < r. Denote by Az, the 
ratio of the maximum-likelihood given Hx (that the chain is of order k) 
to that given H, (that the chain is of order r); then we get 
№, Pe А.к paik + Arar 
and so kA, — = 2 log Ak з= 


e —2dlogAea К< г, 
Assume the variables — 2 log A, 


vr (070, 1, ...) to be asymptotically 
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independent, given Ну; then it follows that 44, has 2-distribution 
with d.f. 
уз з уз a k > 0 


and vsrt! , К=—1, 
where VS PI. px. 


Now the question of choice of an appropriate loss function arises 
once this identification procedure is considered as a decision procedure. 
The loss functions considered in classical theory of hypothesis testing are 
defined by the probabilities of accepting the incorrect or rejecting the 
correct hypothesis. 

Tong proposes the choice of the loss function, based on the AIC 
approach as 


R(k) — кА» =p! (vst T ysti), 


where m is the highest order model to be considered and k is the order 
of the fitting model. The MAICE of the best approximating order of 
the Markov chain is that value of k which gives the minimum of R(k) 
over all orders considered. Note that A(m) = 0. 

Gabriel and Neumann described the occurrences and non-occurrences 
of rainfali (of Tel Aviv) by a two-state Markov-chain. A dry date is 
denoted by state 0 and a wet date by state 1. In the following example 
we consider a similat model for weather data (Medhi, 1976). 


Example 6(a). The following data are obtained from observations of 
the occurrence and non-occurrence of rainfall in Gauhati, India. A dry 
date is denoted by state 0 and a wet date by state 1. The period covered 
is from the last day of February to the last day of May (i.e. before the 
onset of monsoon in early June) for four consecutive years 1967-70 and 
N, the total number of transitions in the 4 years is 4 x 92 — 368. The 
transitions are as noted below. 


zi 0 1 Total 
0 175 49 224 
1 48 96 144 
Total 223 145 368 
The estimated transition matrix P — (р;;) is given by 
0 1 
0 m wa 
1 1/3 2/3 ]* 


We have 
Pie Wut un 


0.59289 040711 
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0.60390 0.39610 po [e 60377 aped 
pue Pape 0.39642/^ ~ 0.60377 0.39623): 
Using (4.6), we get lim P" — (0.60377, 0.39623). 
ЕР no 


We wish to test the null hypothesis that the chain is of order zero 
against the alternative that it is of order one. 


Hopp, hj-i2. 


We have 
2252 3 m lena 
Reka 5 щи ортосу, 
д „ (368) (175) , (368) (49) (368) (48) 
=ofi75 Іов (13) (24; 4° 18 as) (223) + 48 10092) (144) 


(368) (96) 
+ 961ов тс) у 


== 53.28. 


For 1 d.f., Pr {0 > 53.28} being very very small the null hypothesis 
that the chain is of order zero is rejected. Again, we wish to test the 
null hypothesis that the transition probability matrix is P against the 
alternative P? = (22), 


where pa. s 127/19 8/15 

Hir BP = ine 4/5 
is the transition probability matrix of dry and wet days during the mon- 
soon months (June-August). 


We have 
2 2 ;— pe 
5 х MP Pu _ 195.07, 
i=1 j=1 P ij 
For 2 d.f., Pr {x? > 105.07} being extremely smali, the null hypothesis 
that Hy: P= P” is rejected. 


. 3.7 Markov Chains with Continuous State Space 


So far we have discussed Marov chains {Xn n = 0, 1, 2,...} wlith dis- 
crete state space, i.e. with 0, + 1, + 2,... as possible values of Y,. Here 
we consider chains {Х„} with continuous state space, i.e. with (— co, со) as 
possible range of values of Х„. We shall have to use either probability 
distribution function (d.f.). or probability density function (p.d.f.), when 
this exists, in place of probability mass function. 


Definition. If, for all m, and for all possible values of {Xm} in (— co, co) 
Pr Xo, < x| Xn — y, Xo = yp. -Xo = Yu] 


= Pr{Xmty < x| Xn = y) (7.1) 
then (X4) is said to constitute a Markov chain with continuous state 
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space. If the conditional d.f. as given by (7.1) is independent of т, then 
the chain is homogeneous; (7.1) gives (one-step) transition probability 
d.f. More generally, the n-step transition probability d.f. is defined by 


Pr {Xmen < X | Xm = y, -> Xo = Ym} 
= Рг (Xnin S x| Xn = » (7.2) 


= Р, (у; х). 


Denote s A 
P (y; x) = P, (у;х) = Pr {Xnr < X | Xm = у} = Pr {Xn < X | X,=y}- 
Let P, (x) = Pr (Xn < х}. The transition d.f. Р, (y; х) and the initial 
distribution Pr (X, < х} = Ро (х) can unquiely determine Р, (х). The 
Chapman-Kolmogorov equation takes the form 


Prim (Ухх) = J dzPn (у; 2) Pm (23%): m,n > 0, (7.3) 


which corresponds to pg+™ = Z pi pt for Markov chains with discrete 
s 


state space. 
For m = 1, we can write (7.3) as 


Pray (5 2) = | & Po 5 2) 659 


= | d Рф; 2) РС). (7.4) 
Suppose that as n > ©, Pn (у; x) tends to a limit P (x) independent of 
the initial value. Then the limiting distribution P (x) satisfies the 
integral equation 
P (x) = | de P(G)P (ex) 


Ш 


| P (z; x) dP (z). (7.5) 


The relation (7.5) corresponds to (5.2) for chains with discrete state 
space. In the above relations distribution functions can be replaced 
by p.d.f.’s when these exist. It is to be noted that the theory of Markov 
chains with continuous state space is not so fully developed as that of 
«chains with discrete state space. Special methods are however developed 
to study particular types of chains such as random walks. We consider 


here two examples. 


Example 7 (a) Random walk: Suppose that the steps of the unrestricted 
random walk are 1.1.4. continuous random variables Z,, Z.,... with a 
p.d.f. f, — со < 2 < соо. Wehave Х, = Z, + ... + Zn so that Xn4. = 
Х,а: Clearly, {Xn} is a Markov chain with a continuous state space. 


The (unit-step) transition d.f. can be written as 
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P(y; х) = Pr {Xna < x | Xn = y) 


= Pr {Xn Zna S x| Xn S у} 
whence we get 


Дх) = [ron X — 2) dz, 


where /, (2) is the n-fold convolution of f(z). Here f(x — z) which is 
the p.d.f. of X54, == X, + Zn4, given that Y, = z, is the transition p.d.f. 


Example 7 (b). Queueing process*: Suppose that customers arrive at a 
facility at instants žo ( == 0), 1, ty)... i.e. an == tna, — t, is the interarrival 
service time between the ath and (л -:- 1) st customers. Suppose that 
bo, Бу, bas... are the durations of the successive service times. Assume 
that ал, b, are sequences of independent, positive random variables with 
d.f. A(x) = Pr {an < x) and B(x) = Pr (b, < x} respectively. Let the 
random variable W,, = W(t,—0), denote the waiting time of the nth arrival. 
Let X, = b, — an have d.f. F(x) = Pr {Xn < xk 
We have, for n 20, 1, 2,... 


Ил Wat Xa if И, ЕХ, > 0 
= 0, if Wyt+Xn <0. (7.6) 


It is clear that {Wn} constitutes a Markov chain with (0, со) as possible 
range of its values. We have 


PQ(v;x)-—0, if x «y 
=1, 1 хру 
апа Pa (y; х) = Pr{W min < х | Wa =} for 2 = 1,2... 
In particular, the unit-step transition d.f, is 
Р(у; x) = Р,(у; х) = Pr (os, < x | Wa = y) 
—Pr (W4,,—0 | W,:y)-- Pr (0 < Was, <x | Wy) 
= Pr {y+ b — a, < 0)--Pr (0 < y--b,—a, < x} 
—Pr(ytb—azx) * 
= Pr(b, — an < x — y) 
=F(x— у). (7.7) 
From (7.4), we get, for n —0, 1, 2,... 


Dn Vine f Р(х — z) d; Р„(у;2) (7.8) 


= F(x—z) P, (у; z) |z2o -Í dF (x — z) Р, (у; 2) 
° 


*For a description see Chapter 10. 
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= - [ 05 dF(x — z). (7.9) 
0 


Suppose that lim Р, (у; =) = P (z) exists, then we get 
no 


P(x)= f F(x — 3) dP (2) 
= [ P(z) dF (x — 2). (7.10) 
0 


3.8 -Non-stationary or Non-homogeneous Chains 


The Markov chains considered so far are time-homogeneous or station- 
ary. In other words, we have assumed that for a discrete space Markov 
chain (X,), the transition probabilities p;; — Pr {Xn 5j | Xn- = i) (and 
therefore the corresponding t.p.m. P = (p;;)) are independent of n. This 
assumption leads to some elegant results, as already discussed. However, 
the model of a homogeneous or stationary chain is not appropriate in many 
real situations arising, for example, in epidemiology, learning theory, etc, 
This points to the necessity of considering also non-stationary chains, 
which have so long received little attention in literature at least as much 
as the general case is concerned. 

We consider here a matrix approach developed by Harary et al. (1970) 
for finite non homogeneous chain. This makes use of the notion of a 
causative matrix that, when multiplies a t.p.m. yields a subsequent one. 

Suppose that the transition probabilities p;; are functions of n, i.e. 


Pr (X, — | Xia = i) = ру (п). 
Then the probabilistic situation is described by a sequence of t.p.m.'s 
Р Раа: Рав 


In the case of homogeneous chains, the situation is described by а single 
tp.m. P, іе. Р, = P for all п А sequence of causative matrices 


Cy, C,.. Cn... are introduced to describe the change in the transition 
probability matrices. These causative matrices will be defined by 
P,C, = P; PC, = Р„..., Р.С = Рау... (8.1) 


From (8.1) we get Cn = P; Ра, provided the matrix P, is nonsingular. 
The matrices C, multiply Р, from the right to yield Ру. Likewise one 
may consider causative matrices which involve multiplication from the 


left. 
. The homogeneous chain can be considered as a special case obtained 


by taking 
С„= forallm: 
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We shall assume that all the causative matrices C, are equal, i.c. 


C, = C for all n. We then have 


Pua = В,С" (8.2) 
and, in general, 
Prim = Py, Ce», n=0, 1025 ха 


Further, if the matrices are non-singular, then 
Pry = Pa Pra Р, 
and Py = (Р,Рг?)"-° Р, = P, (PL Py, п=2,3,.... 


Denote the t.p.m from period u to v by P(u, v) and P, = P (0, 1) by Q. 
Then 


Pin, n+ D= P, 
=PO й=0,1,2 +. 


апа P(0,n + 1) = P(0, 1) Р(1,2).. - P(n, n 4- 1) 
= Q. QC....Qc^ | 
s | 
= П(0С) = Т, (sy). (8.3) 


А good deal of information about а chain can be had from the 
matrix C itself. The limiting Properties of a constant chain (with 
С, = C, =... = C) depend on the convergence of С" and Т, as n— со. 

First we consider the following result: 


Theorem 3.8. If Q and R are two stochastic matrices of the same 


order and if Q is non-singular, then the causative matrix C — QIR has 
unit row sums. 


Proof: Тебе = (1, 1, ..., 1)’ be a column vector with unity as each of 
its elements. Then Re — e, Oe — e and hence Q-e-—e, Thus 
Ce = Q7! Re = 0-1е = e, 


Note that C can have negative elements and so C may not be a stocha- 
stic matrix. 


Some of the relevant questions are the existence of the limit and 
stochasticity of C, and Ta. We examine them for the two-state non- 
stationary chains (for details, see Harary et al). 


Two-state non-stationary chains 
Let Р, = О and P, = К be given by 


а 1—a EM E 1,—;ё 
hos b кады гел 2 


and the causative matrix be 


u r= й 
= 8.4 
© (ea eee eD 
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C isa stochastic matrix iff 


О<н=<1,0<пини—=т=<1, 


ie. max (0, т) < u < min (1, 1+ т). (8.5) 
Since 
Qu PtP 
we have 


u= (bc — (1 — a) (1 — d)f(a + b— 1) 
= (a + d — ad + bc — 1)/(а -- b — 1), 
and m = (c 4- d — 1)/(а + b — 1). 


The characteristic roots of C are 1 and m, so that the characteristic 
roots of С" are 1 and m”. Since f,(C") = sum of the characteristic roots, 
and C" has unit row sums, we may write 


© ип T= i k 
emo от МНЕ ладна 


taking ш =u. Since С" = С.С" = С".С, we have 
Ung = Un — т" 4- um" =u — m+ mus n = 1, 2,.... 
Solving this difference equation of order 1, we get un. For m = 1, 


Ипа = Un и 1 


= Un + 2(u— 1) 
=u +n(u— 1) 
or, иһ =1+n(u—1) (since u, = u). 


For m # 1, we get 
Un = {u — m + т" (1 — uy — т). 
Thus, we get 


lim и te mul 
Logs d: и — т)/(1— т), —1<т<1. 


The limit does not exist otherwise. Thus, form=u=1. 
А 1 0 
a= 

inre if i 
When т = и = 1, the causative matrix Creduces to the identity matrix 
and the chain becomes homogeneous. 

For —l<m<l, 
lim C" = (и = т)/(1— т) (1—и)/(1 i) 
= Xu ma — т) (1 — uf — m) 


mo 
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= ((и— т)/(1 — m), (1— и)/(1— т)). (8.6) 

The limiting matrix is stochastic iff 

0 x (u— т)/(1— т) < 1 
ог ти 1; (8-7) 
when 0< m< 1, then (8.5) and (8.7) are equivalent, i.e. lim C" is 
stochastic iff C is stochastic. 

More important are the existence and stochasticity of 

lim 7,= lim П (QC?) 

no n r=0 
These are considered in the following theorem. 


Theorem 3.9. Let m be the non-unit characteristic root of the causa- 
tive matrix C. 


If —l1<m<1, lim T, exists and is equal to 
п = 


((и — т)/(1 — т), (1 — i — m). 


If 0 x m — 1, lim T, is stochastic iff C is stochastic. . 
п->= 


If —1<m<0, QC is stochastic for all r = 0, 1, 2, 3,... and so are 


T, and lim Т„, but then C is stochastic provided only 0 < u < 1 + m. 
п ә 


For the proof which involves algebra of stochastic matrices refer to 
Harary et al. 

The analysis of non-stationary chains provides interesting insight into 
problems otherwise intractable and also furnishes basis for useful models 
for many situations in real life. 

The elegant result concerning the asymptotic behaviour of the n-step 
transition probabilities of homogeneous Markov chains does not hold 
good in case of non-homogeneous chains. For some recent results, on 
such behaviour of non-homogeneous chains, réference may be made tpa 
paper by Cohn (1976), who presents a general treatment of the asymp- 
totic behaviour of the transition probabilities between various groups of 
States. For an account of the historical development of the theory of 
non-homogeneous chains, see E. Seneta (1973). For recent contributions 


refer to Kingman, (1975), Cohn (1976), Isaacson and Madsen (1976), etc. 
(see also Paz (1971) for an account), 


EXERCISES* 


3.1 A communication system transmits the two digits 0 and 1, each of 
them passing through several stages. Suppose that the probability 


*P denotes the transition 


Probability matrix of the Markov chain referred to in 
the exercise. 
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that the digit which enters remains unchanged when it leaves, is p 
and that it changes is у -1— p. Suppose further that X% is the 
digit which enters the first stage of the system and X,(n > 1) is the 
digit 0 leaving the nth stage of the system, Show that {¥,} forms a 
Markov chain. Find P, P?, P? and calculate 


Pr(X,—0]|Xo— 1) and Pr(X,—1]| Х, = 0). 


3.2 Three children (denoted by |, 2, 3) arranged in a circle play a game 


3.3 


3.4 


3.5 


3.6 


3.8 


3.9 


3.10 


of throwing a ball to one another. Ateach stage the child having 
the ball is equally likely to throw it into any one of the other two 
children. Suppose that X, denotes the child who had the ball initially 
and X,(n > 1) denotes the child who had the ball after п throws. 
Show that {X,} forms a Markov chain. Find P. Calculate 


Pr(X,—1| X, = 1), Pr (X, 22| X, = 3), Pr(X, =3 | X, = 2), 


and also the probability that the child who had originally the ball 
will have it after 2 throws. 

Find P if the number of children is m( > 3). 

Consider a sequence {X,} of independent coin-tossing trials with 
probability p for head (H) in a trial. Denote the states of X, by 
states 1, 2, 3, 4, according as the trial numbers (n — 1) and п result in 
HH, HT, TH, TT respectively. Show that {X,} isa Markov chain. 
Find P; show that P"(m > 2) is a matrix with all the four rows 
having the same elements. А 

Suppose that a fair die is tossed. Let the states of Y, be k(=1,2...., 6), 
where К is the maximum number shown in the first п tosses. Find 
P, P? and Р". Calculate Pr (Y, = 6). 

Suppose that a ball is thrown at random to one of the 3 cells. Let 
X,(n > 1) be said to be in state k( = 1, 2, 3) if after n throws К cells 
are occupied. Find P and P". 

Occupancy problem (Generalisation of Exercise 5): Suppose thata 
ball is thrown at random to one of the r cells. Let X,( > 1) be said 
to be in state k( = 1, 2, ..., r) if after n throws k cells are occupied. 
Find P and P?, ч 

Find out whether the Markov chains having for transition matrices 
those given in Exercise 1.22 are irreducible. 

A coin is tossed, p being the probability of head in a toss. Let Y, 
have the two states 0, 1 according as the accumulated number of 
heads and tails are equal or unequal. Show that the states are 
transient when pÆ $, and persistent null when р = 1. 

A particle starting from the origin moves from position j to posi- 
tion (j+ 1) with probability a; and returns to origin with probability 
(1 — aj). Suppose that the states, after n moves are, 0, 1, 2... . Show 
that the state 0 is recurrent iff lim (d, .... a5) 0 as n > оо. 
Suppose that for a Markov chain pj, j..=v;, pj,>=1—v; and p;,,—0, 
к=] + 2, 0. Show that the state O(Y, is at the origin after n moves) 


9z 


3.11 


3.12 


3.13 


3.14 


3.15 
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is recurrent iff (vy. . Vn) > 0 and non-null iff ZẸa. . .yy,-,)isfinite. In 
п 


particular, when у; = 6% (a> 0), state 0 is recurrent non-null. 
(In Exercises 9 and 10, the random walk is non-homogencous.) 
Show that, for a chain with a finite number, m, of states and having 
a doubly stochastic matrix (ру) for its transition matrix, 

Did ve = Mm for all j, k= 1,2, ...,m. 
(A non-negative square matrix is said to be doubly stochastic if all 
the row and column sums are unity.) 
A necessary and sufficient condition that all the states of an irreducible 
and aperiodic Markov chain possessing a finite number of states are 
equally likely in the limit is that the transition probability matrix is 
doubly stochastic. 
A Markov chain with states 0, 1,... has transition probabilities 


х (i 
Dik = е7 5 ( \„ (1 — py-'rak-r[(k — r)!, 
тыр V 


Ж ESO) 1,2125 Odp ET 


Show that the limiting distribution {тк} is Poisson with parameter 
a/(1 — р). ` 

Grouping of states of a Markov chain: Consider that three states of 
the Markov chain {X,} described in Exercise 3 above are grouped 
together as follows: У, is said to be in state 0 if both the (n — 1)th 


and nth tosses result in Heads and 1 otherwise, (i.e. state 0 for HH 
and state 1 for HT, TH, TT). 

Show that the chain {Y,} is non-Markovian. 

If the states of a Markov chain are grouped or lumped together, the 
new chain does not, in general, have the Markov property. 
Inventory model with (s, S) policy: Suppose that the stock position 
of an establishment is examined at epochs of time t,, t}... and the 
following policy is adopted. If at epoch fy, the stock level is less than 
or equal to s, procurement is immediately done te bring it to the level 
S and if the level is greater than s, no replenishment 1s done. It is 


supposed that the demand a, during (t, ,, fn) obays the probability 
law 


Pr (аһ = К) = vx, b Ук== 1; 
k=0 


and that {an} are i.i.d. random variables. Suppose that the states 
of {Xn} consist of possible values of stock size just before examina- 
tion at epoch f, i.e. states are 5, S—1,..., 0, —1, —2, .. ., negative 
values being interpreted as unfulfilled demand for stock to be fulfilled 
as soon as replenishment is done. Show that (Y,, п > 1} is a Markov 
chain. ў 


Branching process (with one type of organism ог particle): Suppose 
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3.16 


that at the end of a lifetime an organism produces similar offspring 
(or a particle at the end ofa transformation gives rise to similar new 
particles) with probability Pr(Z -= i) = pi, 1—0,1;2,..5 2 pros T. 
i 
Suppose that the production of offspring does not depend on the 
evolution of the process and that an offspring produces new off- 
spring, independently of others and in the same way as their parents. 
Let {Xn} denote the population size at the nth generation (п > 1). 
Assume that X, — 1, i.e. the process starts with one ancestor. Show 
that (X,) is a Markov chain. 
Let 


рд) = X Pr(Y—j) s 
j=0 
be the p.g.f. of Xn and 
P(s) = Ps) — E Pr (X, =j) j= X ру! 
j-0 j=0 


be the p.g.f. of X,- 
(P(s) is also the p.g.f. of the number of offsprings per individual.) 


Show that 
(Q Pau) = Pa(P(s)) = Pn—s(P2(s)) =... = РСРл(5)). 
(ii) рук == coefficient of s* in (P(s))/. 
(iii) pi") == coefficient of sk in (Pa(s)). 


A detailed account is given in Chapter 9. 

(See also Athreya and Ney, 1972.) 

Consumer Brand switching model (Fourt, Whitaker) : Consider that 

there are N brands of a consumer product competing in a market. 

Define 

d; = brand loyalty for the ith brand (equal to the proportion of 

consumers repurchasing the same brand i without persuation 
next time), 

w; = purchasing pressure for the jth brand (equal to the proportion 
of consumers who are persuaded to purchase the jth brand 
next time). 

Then the brand switching transition matrix is given by А = (4), 


where 
ay = di ++ (1 — di) wj, j=ij=1,.. N 
=(1— 4)», Usi = lyar Ni 
N 
0<4<1,0< и; <1, 2 и = 1. 
j=l 
Thus X 


ay gives the proportion of customers а, remaining with the ith 
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brand plus the proportion of disloyal customers (1 — d) who 
remain with the same brand after being persuaded by the pur- 
chasing pressure w. 

dj; (Æi) gives the proportion of disloyal customers who switch 
from /th to jth brand being subjected to purchasing pressure w;. 
‘The brand share of th brand in period г is. given by the ele- 
ments of л 


where po Vey Ai 


Show that the equilibrium brand shares v; (equilibrium prob- 
abilities v;) are given bv 


е 
Y wl dj) [3 (vl — din. I. as eer 


Examine the particular cases (i) d; — d : (ii) d- 1. 

Whitaker (1978) develops a technique of estimating brand loyalty 
by using least squares method to fit the above Markov model to 
aggregate data. 
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CHAPTER 4 


MARKOV PROCESSES WITH DISCRETE 
STATE SPACE: POISSON PROCESS 
AND ITS EXTENSIONS 


4.1 Poisson Process 


A. Introduction 

Here we shall deal with some stochastic processes in continuous time 
and with discrete state space, which, apart from their theoretical 
importance, play an important role in the study of a large number of 
phenomena. One such process is Poisson process to which we shall 
confine ourselves here. To fix our ideas, consider a random event E such as 
(i) incoming telephone calls (at a switchboard), (ii) arrival of customers for 
service (at a counter), (iii) occurrence of accidents (at a certain place). 
Let us consider the total number N(t) of occurrences of the event E inan 
interval of duration /, i.e. if we start from an initial epoch (or instant) 
1—0, N(t) will denote the number of occurrences upto the epoch 
(or instant) ¢ (more precisely to ¢ -- 0). For example, if an event actually 
occurs at instants of time /, fy, f}... then A(r) jumps abruptly from 
Oto 1at = г, from 1 to 2а1 = t, and so on; the situation can be 
represented graphically as in Fig. 4.1. 


Nit) 


th f t 
з а 
t 5 


Fig. 4.1 Number N(t) of occurrences of an event in an interval of duration (0, £) 


The values of Мг) given here are observed values of the random variable 


N(t). Let pa(t) be the probability that the random variable N(t) assumes 
the value л. i.e. 


MARKOV PROCESSES WITH DISCRETE STATE SPACE 97 


Palt) — Pr {N(t) = п}. (1.1) 


This probability is a function of “the time /. Since the only possible 
values of n aren = 041, 2. 3,.... 


2 Pat) — 1: (1.2) 
thus, {p,(t)} represents the probability distribution of the random variable 
M(t) for every value of t. The family of random variables {N(1), t > 0} 
is a stochastic process. Here the time / is continuous, the state space 
of M(t) is discrete and integral-valued and the process is integral-valued. 
(such a process we also call a counting process). One of the most 
important integral-valued processes is Poisson process, which serves as 
a mathematical model for a wide range of empirical phenomena with 
remarkable accuracy. The justification for this is based on the concept 
of rare events. 

We proceed to show that, under certain conditions, М?) follows 
Poisson distribution with mean Az (А being a constant). In case of 
many empirical phenomena, these conditions are approximately true and 
the corresponding stochastic process {N(t)} follows the Poisson law. 


B. Postulates for Poisson Process 

I. Independence: МІ) is independent of the number of occurrences 
(of the event E) in an interval prior to the interval (0,7). ie. future 
changes in №(/) are independent of the past changes. 

П. Homogeneity in time: pa(t) depends only on the length z of the 
interval and is independent of where this interval is situated, ie. pa (1) 
gives the probability of the number of occurrences (of /:) in the interval 
(ty t+ гу) (which is of length /) for every 1,. 

ШІ. Regularity: In an interval of infinitesimal length л, the probabi- 
lity of exactly one occurrence is Ah !- o(/) and that of more than one 
occurrence is of о(/). (Occurrences are regular or orderly.) 

o(h) is used as a symbol to denote a function of Л which tends to 0 
more rapidly than А, i.e. 


as h — 0, ша (1.3) 


In other words, if the interval between г ands Ais of short duration 
h, then 


ру (А) = M -} о(һ) (1.4) 
$ Pe (h) = о(А) (55) 
Since m 


X piak 


n=0 


it follows that 
po (А) = 1 — АА - oh). (1.6) 


7 (45-95/1982) 
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Theorem 4.1. Under the postulates I. IT, IIT. N(r) follows Poisson distri- 


bution with mean Az, i.c. p, (t) is given by the Poisson law: 


3s (fj = E п 015.80. (1.7) 


Proof: Consider p, (t ` h) forn > 0 


0 1 t+h 
n events by epoch г + A can happen in the following mutually exclusive 
ways 41. Ao, Aa. - -Anga ’ 
Fornèl 
A,: п events by epoch / and no event between f and t + h; 
we have 
Pr (A,) = Pr {N(t) = nj Pr {N(h) = 0 | M(t) = n} (1.8) 
= Pa (t) Po (h) 
— pa (1) (1 А) + o(h) 
А: (п — 1) events by epoch z and І event between t and f - Л; 
we have 
Pr (Ay) = Pr {N(t) = n — 1} Pr {N(h) = 1 | МИ) = п – 1} (1.9) 
Pa- (t) р, (h) 
= Pn (t) (Ай) ~ o(h) 


Forn>2 
Аз: (п — 2) events by epoch гапа 2 events between гапа г + h; 
we have Pr (As) ра, (0) {o(h)} = o(h) 


And so on for Ay, A; etc. We thus have Pr (Ax) = o(h) for n > k > 3. 
Adding the probabilities for these mutually exclusive events, 
we have 


Pa (t 1 h) = Pa (t) (1 — А) + paz (0) (АЛ) + ofh) n 21 (1.10) 


n (t +- h) — pa (t 
RE; en A Р о. — Арп (1) + Apn- (t) + a) 


In the limit, as л > 0 
Dy (t) = —A[Pa(t) — Poa (@)], п > 1 (1.11) 
For n = 0, we get 
Py (t 4- AY = Pa (t) Po (A) = Po (0 (1 — Ah) + ofh) 
Palt | 1) — py (t) 
КыЛ). — Aper) 4 


or, 


olh) 
h 


whence, as л > 0 


Po (1) = — Apa (t) (1.12) 
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Initial condition: Suppose that the process starts from scratch at 
time 0 (or at the origin of epoch measurement), so that A(0) — 0, i.e. 


pO) — 1: p«(0) 0 for n #0. (1.13) 


The differential-difference equations (1.11) and (1.12) together with 
(1.13) completely specify the system. Their solutions give the proba- 
bility distribution pt) of №). The solutions are given by 

рй = с eee ODE. (1.14) 
(see Example 1.6(b) where the equations were considered to illustrate the 
use of Laplace transform in solving differential-difference equations). 

We indicate here two other methods of solving these equations. 

(1) The method of induction : The solution of (1.12) is given by 
palt) = Се-М. Since p,(0)-=1, we have C =: 1 so that рог) = e-M, Consider 
(1.11) for n = 1. Substituting the value of p, and solving the equation 
and using (1.13), we find pit) = M. e. Thus (1. 14) is seen to hold for 
п== 0 апа 1. Assuming that it holds for (n — 1) it can be shown like 
wise that it holds for n. Hence, by induction, we get (1.14) for all л. 

(2 The method of generating function: Define the probability 


generating function 


P(s, 1) = H ра)" = E Pr {М0О- пу" — E{sN@}, (1.15) 


Now Pos, 0) = z p(0)5" = р(0) + р(0)з +... = 1. (1.16) 
We have 

ê P = 5 д Js” z I 4 п 1 Ц . 

s 6-2 sn (0s" + pot); 


i Pa(t)s" = P(s, t) — Pt); 


and 
P Pn (57 = sP(s,t). (1.17) 
nel 

Multiplying (1.11) by 5" and adding over for n= 1, 2, 3,. ..and using 


(1.17), we get 
z Р, 1) — pol(t) = — MEG, 1) — p) — sFG, 0] 


or 


Thus 


z рз. D) Ant) = PG, DIG — D) + АРМ). 


E P(s, t) = P ($, HAS — D} (1.18) 
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Solving (1.18), we get 
Ds; T) — ARC US (1.19) 
Now P(s, 0) — 1 from (1.16), so that A — I. 
Hence the p.g.f. of Poisson process is given by 
P(s, 1) = сі! 6-0 (1.20) 
= ы! х aa. . (1.21) 


ney N! 
so that 
Palt) = coefficient of s? in P(s, 1) 


=À 
ECAN, nO! (1.22) 
п! 
Corollary 1. We have 
E{N(t)} — Mt (1.23) 
and var {N(t)} — At. (1.24) 


The mean number of occurrences in an interval of length / is Af, so that 
the mean number of occurrences per unit time (1 — 1), ie. in an interval 
of unit length is А. The mean rate A per unit time is known as the para- 
meter of the Poisson process. 

The mean and the variance of M(t) are dependent on 7; in fact, its 
distribution is functionally dependent on /. As such the process (N(t)) is 
not stationary —it is evolutionary (Sec. 2.3). 


Corollary 2. If E occurred r times upto the initial instant 0 from which 
t is measured, then the initial condition will be 

PAO) -1, p0)-—0, nr. (1.25) 
. Then p,(t) = Pr (Number N(r) of occurrences by epochtisn — r, n > r} 


и eM (At )"-т 
ТЕЧЕ 


nzr (1.26) 


Remarks: 

Postulate I implies that the Poisson process is Markovian; postulate 
II that the Poisson process is time homogeneous; postulate III 
that in an infinitesimal interval of length Л, (ће probability of exactly 
one occurrence is approximately proportional to the length л of that 
interval and that of the simultaneous occurrence of two (or more) events 
is extremely rare. In (1.8) we have put р 


Pr (NU) = 0 | NG) = n) = Pr {N(H) = 0) = palh) 


MARKOV PROCESSES WITH DISCRETE STATE SPACE 101 


(here the two intervals, one of length / and the other of t, are non-over- 
lapping). In general, 
Pr (NI) = k | NO = n) = Pr (NUI) = К} = p(l), k 2 0. 

Poisson process is a Markov process such that the conditional proba- 
bilities are constant (independent of t) and are given by (1.4)-(1.6). We 
shall consider time-dependent Poisson process in Sec. 4.3D. It can be 
easily verified that the Poisson process satisfies the Chapman-Kolmogorov 
equation (5.11) (see Sec. 4.5 and also Example 5(a)). 


Example 1(a). Suppose that customers arrive at a Bank according to a 
Poisson process with a mean rate of a per minute. Then the number of 
customers N(/) arriving in an interval of duration ¢ minutes follows a 
Poisson distribution with mean at. If the rate of arrival is 3 per. minute, 
then in an interval of 2 minutes, the probability that the number of 


customers arriving is 
(1) exactly 4 is 
e-* (6)! 
era 0.133, 


(ii) greater than 4 is 


(iii) less than 4 is 


3 e-* (6)* 
> —E 0.152; 


k=0 


(using tables of Poisson distribution). 


Example 1(b). A machine goes out of order whenever a component (part, 
fails. The failure of this part is in accordance with a Poisson proces: 
with mean rate of 1 per week. 

Then the probability that two weeks have elapsed since the last failure 
is e? = 0.135, being the probability that in time? — 2 weeks, the 
number of occurrences (or failures).is 0. 

Suppose that there are 5 spare parts of the component in an inventor) 
and that the next supply is not due in 10 weeks. The probability tha 
the machine will not be out of order in the next 10 weeks is given by 


5 g3 (10) _ 
2 k! = 0.068, 


being the probability that the number of failures in t = 10 weeks will b 


less than or equal to 5. 


Example 1(c). Estimation of the parameter of Poisson process. For 


Poisson process N(t), as t — © 
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pe{ [MO -a| ae bo. (1.21) 


where = > 0 is а preassigued number. 
This can be proved by applying "Tshebyshev's lemma 
Pr(|X— Б(Х)| >а} <= „Гоги > 0 


From the above, we have, for X = Nit) 


Pr( ND А> 


or pd -a 2e» 

A 
м еее 
Непсе 


Pr {| MO —al def oor o. 


This implies that for large /, the observation N(t)/t may be used as a 
reasonable estimate of the mean rate A of the process {N(t)}. 


C. Properties of Poisson Process 


1. Additive property: Sum of two independent Poisson processes 
is a Poisson process. 


Let N,(f) and N,(t) be two Poisson processes with parameters Ay, № 
respectively and let 


Ки) = N, (t) + №0). 
The p.g.f. of N;(t) (i= 1, 2) is 
E {si} = gni , 
The p.g.f. of №/) is 
E{sNO} = E (sN(-- Nt), 
and because of independence o! N,(t) and iV,(t), we have 
E {sO} = E {5M0} E {Nato} 
= {eM 8200) (e 8-0 
== etA) (5-1) 
Thus N(t) is a Poisson process with parameter A, + А,. 


The result can also be proved as follows : 


Pr{N(t) = п) =} Pr {N,(t) = r} Pr (N41) = n r} 


2$ € Quy er Que 
r=) r! E (n— r)! 
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{ eo Out ae (ON EAM 


an yn. (1.28) 


Hence .N(/) is a Poisson process with parameter (^, + Л). 
2. Difference of two independent Poisson processes: The probability 
distribution of NU) NV, (t) — А0) is given by, B 


A.V? € 
Pr (Ми) =н. о E (=) Tin t V As), n=0, +1, Doe 
(1.29) 


= Cyr n 
where 1(х) = % ш (1.30) 


reo Г!Т(г-ЕП + 1) 
is the modified Ressel function of order n(>—1). 
Proof: (i) The p-g.f of M(t) is 
FK (sNOY |. E (sit) NÀY 
= E (sU) E (s- N40), 
because of the independence of №,(/) and N,(t). Thus 
E (sv) = E {sO} E (1/590) 

= exp fA,f(s — D) exp rs! — Dj) 

— exo (— (A, -- A9) exp {Ats | Agt/s}; (1.31) 
Pr{Nt) ~ n] is given by the coefficient of s” in the expansion of the 


right hand side of (1 .31) as a series in positive and negative powers of s. 
(ii) Pr {N(t) == nj can also be obtained directly as follows: 


Pr (№) =n} = Ў тг (N (G4) =n +r} Pr(NAQ) = г} 


me MA, ttre At) 


„о (n4 r)! r! 


= eon (RT b (УХ Yn 


rao rl(r +n)! 


in, M]? * с 
22 er orb Gy Te ZA 
Tt may be noted that : 
(1) the difference of two independent Poisson processes is mof a 
Poisson process: 
42) La (t) = Int) = I(t), n = 1,2, 3, ... 
(3) the first two moments of M(t) are given by 
E {N} =A, — мг E {N(= Q + Ay)t + Qu — AI. 


Example 1(d). If passengers arrive (singly) at a taxi starid in accordance 
with a Poisson process with parameter A, and taxis arrive in accordance 
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with a Poisson process with parameter A, then N(t) = N,(r) — N,(t) gives 
the excess of passengers over taxis in an interval f. The distribution of 
N(t), i e., Pr{N(t) = п), n —0, + 1, + 2, ... is given by (1.29). The 
mean of N(t) is (4; — Aj)t, which is >, = or < 0 according as A, >, 
or < A,, and var (N(t)) = (^, + A)r. 

3. Decomposition of a Poisson process: А random selection from a 
Poisson process yields a Poisson process. Suppose that Nt), the number 
of occurrences of an event Е in an interval of length t isa Poisson process 
with parameter A. Suppose also that each occurrence of E has a constant 
probability p of being recorded, and that the recording of an occurrence 
is independent of that of other occurrences and also of N(t). 

If M(t) is the number of occurrences recorded in an interval of 
length ¢, then M(/) is also a Poisson process with parameter Ap. 


Proof: The event {M(t) = п} can happen in the following mutually 
exclusive ways : 


A, : Е occurs (+r) times by epoch г and exactly out. of (n + ғ) 
occurrences are recorded, probability of each occurrence 
recorded being p, (r = 0, 1, 2, .. a) 

We have 


Pr (4,) = Pr {E occurs (n+ г) times by epoch ¢}.Pr (n occur- 


rences are recorded given that the number of occurrences is 
ndr) 


ем (Ayr n+r а 
Шил ( п re. 
Hence 


Pr {M(t) =n} = È Pr (4) 
r=0 


sd SEE ату ag 
rao (+r)! n )ЁЧ 
aeons (Apt)" (Aqt)' 
r=0 nl rl! 
= en Ort у (ду 
AY yen’ TFI 
— „—м Opt e^t (A pt)s 
cer gy C rS ok (1.32) 

We can interpret the above as follows : 

For a Poisson process №(/), the probability of an Occurrence in an 
infinitesimal interval ^ is (approximately) proportional to the length of 
the interval /, the constant of proportionality being A. 

Now for M(t), the probability of a recording in the interval his 
proportional to the length Л, the constant of proportionality being Ap. 
Thus {М(т\} is a Poisson process with parameter А p. 

4. (Continuation of property 3) : The number M(t) of occurren- 
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ces not recorded is also a Poisson process with parameter Ag = A (1 — р) 
and M(t) and M,(t) are independent. 

Thus by random selection a Poisson process (N(t)) of parameter A is 
decomposed into two independent Poisson processes (M(/)) and (M(0) 
with parameters A p and A (1 — p) respectively. 


Fig. 4 2 


As an example, suppose that the births (in a certain town hospital) 
occur in accordance with a Poisson process with parameter A. If the 
probability that an individual born is male is p, then the male births 
form a Poisson process with parameter Ap and the female births form 
an independent Poisson process with parameter А (1 — р). 

More generally, a Poissón process N(/) with parameter A may be 
decomposed into r streams of Poisson processes. If p; Pe ..., pr, are 
the probability of the process decomposing into r independent streams 
such that p, +... + p, = 1 then the Poisson process is decomposed into 
r independent Poisson processes with parameters A pi, А ps, ...,А pr. 

5. Poisson process and binomial distribution : If N(t) is a Poisson 
process and s < t, then 


Pr (N(s) = k | Nt) = п} = (2) (s/t)* (1 — (s/t))"-*. 


Proof: | Pr(N()—k| М) = n} = MAE 


. Pr(N(s) =k and Mt — 5) -n— k} 
» Pr {M(t) = n) 


_ Pr {M(s) =k} Pr (N(t- 5) =r- Kj 
Pr (N(t) = п} 
es (As)* Q3) eae (At)? 


urs 7 руч (п — К)! п! 
п! sk (t — s)" 
k! (n — К)! i? 
= (. ) (вре) (1 — (st). (1.33) 


6. If {1(1)} is a Poisson process then the (correlation) auto-correlation 
coefficient between N(t) and N(t -+ s) is {0/02 + 5). 
Proof: Let А be the parameter of the process: then 
E(N(T)) — АТ, var (N(T)) — AT, 


and E(NNT)) = AT + (AT)? for f= and / —- s. 
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Since M(t) and N(t + s) — N(t) are independent, {N(t)} being a Poisson 


process, 


AMONG © s) = EINN + 5) — NO) -+ NOH 
== E[NQON() + МОРАМЕ + 5) -- NOY 
- EWNAD} + EUNGOVEUNG - s) — MO}. 
Hence E NONU + 5)) -= QUU А?) ALAS. 
Thus the ашосоуагіапсе between A(t) and M(t — s) is given by 
C( t. s): EUNQ)N(- 5)}— ELN(OJELN(t + 5)} 
(Ati Aft? + ONIS) — AAt -- As) — M. 
Hence the autocorrelation function 


- X Ct, t-s) 
PUE sym {var M(t) var NG F s)ju£ 
= uuo se. | (1.34) 
It can be shown that (4, 1") — jus о, (1.35) 


This is the autocorrelation function of the process, 


Example (е). A radioactive source emits particles at a rate of 5 per 
minute in accordance with a Poisson process. Each particle emitted 
has a probability 0.6 of being recorded. The number A(t) of particles 
recorded in an interval of length ¢ is a Poisson process with rate 5 x 0.6 
- 3 per minute, і.е. with mean 3¢ and variance 34. In a 4-minute 
interval the probability that the number of particles recorded is 10 is 
equal to e713(12)/10! —0. 104. 


Example i(f). A person enlists subscriptions to a magazine, the number 
enlisted being a Poisson process with mean rate 6 per day. Subscribers 
may subscribe for 1 or 2 years independently of one another with res- 
pective probabilities 5 and 4. 

The number of subscribers N(/) enrolled by the person in time ¢ days 
is а Poisson process with mean rate 6f: the number N,(t) enrolled for 
l year period is a Poisson process with mean (6 .. 2)/ -- 4t and the 
number N,(¢) enrolled for 2-year period is a Poisson process with mean 
(6 a. 41 -- 21 

If he received commission @ for a l-year subscription and b fora 
2-усаг one, then the total commission earned in period / is given by 


X(t) -aN (t) -- bN (t). 


We have 
E{X(t)} — аЕ м) + БЕ{М,(1)} 
4at `- 2bt 
and var Хп) 4t 2р2. 


Note: The process XU) is a compound Poisson process (sce Sec. 4.3B). 
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4.2. Poisson Process and Related Distributions 


A. Interacrival Time 

With a Poisson process, М} the number of occurrences of an event E 
by epoch z, there is associated а ;aadont variable--the interval X between 
two successive occurrences of E. We proceed to show that Y has a 
negative exponential distribution. 


Theorem 4.2. The interval between two successive occurrences of a 
Poisson process {M(t)} having parameter А has a negative exponential dis- 
tribution with mean 1/А. 

Proof: Let X bc the random variable representing the interval 
between two successive occurrences of {N(f)} and let Pr (X < x) — F(x) be 
its distribution function. 

Let us denote two successive events by £; and £;,, and suppose that 


E; occurred at the instant г. Then 
Pr (X > x} — Pr {Eim did not occur in (4, /; | х) given that £; occurred 
at the instant /ij 
-- Pr {Ej did not occur in (ti, t; -- х) М) = i) 
(because of the postulate of independence) 
= Pr (no occurrence takes place in an interval (ti, ti + x) of 
length x | М) - ij 


= Pr (Nx) = 01 M(t) =} = р(х) = e75, x > 0. (2.1) 


Since i is arbitrary, we have for the interval Y between any tivo succes- 
sive occurrences, 
F(x) = Pr {X < xp}—1—Pe{X > xj-1— едх, x 2 0. 


The density function is 
f(x) = F(x) = Ает, (x> 0). A (2:2) 


It can further be proved that if X; denotes theinterval between Е; and 
Ena i= 1, 2... ., then X, Vy... arealso independent. We omit the proof 
which is outside the scope of this book. We state the result as follows: 


Theorem 4.3. Тһе intervals between successive occurrences (called inter- 
arrival times) of a Poisson process (with mean At) are identically and 
independently distributed random variables which follow the negative 
exponential law with mean 1/A. 

The converse also holds; this is given in Theorem 4.4 below. These 
two theorems give a characterisation of the Poisson process. 


Theorem 4.4. 1 the intervals between successive occurrences of an event 
E are independently distributed witha conimon exponential distribution 


with mean 1/A, then the events E form a Poisson process with mean Af. 
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Proof: Let Z, denote the interval between (л — Dth and nth occur- 
rence ofa process {N(1)} and let the sequence 2,2. HS be independently 
and identically distributed random variables having negative exponential 
distribution with mean I/A. The sum WW, E Zi t.e n is the wait- 
ing time upto the ath occurrence, i.c. the time from the origin to the th 
subsequent occurrence. JV, has a gamma distribution with parameters 
A, n (Theorem 1.10). The p.d.f. g(x) and the distribution function Fy, are 
given respcetively by 


Anx"c eM 
SUE] ue —— "E X50 
ax) T(n) ы 
t 
and Fy, (f) = Pr {Wn < t = [sw dx. 
о 
The events (№) < л} апа (IV, = Z, d- ... +Zn> t) аге equivalent. 


Hence the distribution functions Fy) and /y,(t) satisfy the relation 
Fw) Pr it 1 — PriWso tj 
— 1 — Prin) <n}— d — Pr (NQ) sz (1 — 1} 
21 Pin 1, 
Hence the distribution function of N(/) is given by 
Fyti == 1) = 1 — Жу) 


t 


Anyai е-%х 
25 ^us dx 
LU 
и 


1 
= | = |p- e dy 
1 Tu) b e [| 


0 
© 


1 
ae oe |н ч Йи 
Tu) f ет» dy 
А 


t 


"y eM AL) 


ч (integrating by parts). 
i-0 Js 


Thus the probability law of №) is 


Pall) = PEIN) — nj — Fx a(n) — F(t — 1) 
z GRAID? L'ye7NAty 
i=0 JH i=0 j! 
етм)" 


ү i= 0, 1,2). 
n: 


Thus the process Nq) is a Poisson process with mean At, A 


Note that Poisson process has independent exponentially distributed 
interarrival times and gamma distributed waiting times. 
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Example 2(a). Suppose that customers arrive at a counter in accordance 
with a Poisson process with mean rate of 2 per minute (A = 2/minute). 
Then the interval between any two successive arrivals follows exponential 
distribution with mean 1/A — 3 minute. The probability that the interval 
between two successive arrivals is 


(i) more than 1 minute is 
Pr (X > 1) = e~? = 0.607 
(ii) 4 minutes or less is 
Pr(¥<4)=1- e-^} = 1 — e-2 = 0.865 


(iii) between 1 and 2 minutes is 


2 
Pral<X<2)= f e-*l? dx —0.239. 
1 


Example 2(b). Suppose that customers arrive at a counter independently 
from two different sources. Arrivals occur in accordance with a Poisson 
process with mean rate of A per hour from the first source and џ per 
hour from the second source. Since arrivals at the counter (from what- 
ever source) constitute a Poisson process with mean (A | p) per hour, 
the interval between any two successive arrivals has a negative expon- 
ential distribution with mean f1/(A -! p)} hours. 

For example, if taxis arrive at a spot from the north at the rate of 1 
per minute and from the south at the rate of 2 per minute in accordance 
with two independent Poisson processes, the interval between arrival of 
two taxis has a (negative) exponential distribution with mean } minute. 
The probability that a lone person will have to wait more than a given 
time 7 can be found. 

The Poisson type of occurrences are also called purely random events 
and the Poisson process is called a process of purely random events. 
The reason for this is that the occurrence isequally likely to happen any- 
where in [0, T] given that only one occurrence has taken place in that 


interval. We state this by the following theorem. 


Theorem 4.5. Given that only one occurrence of a Poisson process N(t) 
has occurred by epoch T, then the distribution of the time interval y in 
[0, 7] in which it occurred is uniform in [0, T], i.e. 


Prít <y «t dt| МТ) = у= dT; 0ct«T 
Proof : We have 

Prít« «t4 di) =e dt, 

Pr (N(T) = 1) = e^ QT), 


nd 
Е Pr(N(T) = | у=} = e-XT-0, 
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the last one being the probability that there was no occurrence in the 
interval of length (T — t). 
Hence 
Pr(t <y «t +dt| NT) = 1} 
=Pr{t <y < t 4- dt and N(T) = 1}/Pr (N(T) = 1) 
= Pr{t< y < t + dt} Pr(N(T) = 1] y = tPr(N(T) = 1) 
= Ae Mdt е-е АТ) =adt/T. 
It may be said that a Poisson process distributes points at random over 
the infinite interval [0, co] іп the same way as the uniform distribution 
distributes points at random over a finite interval [a, 5]. 
For a more general form of Theorem 4.5, see Exercise 4.13 (proof is 
similar). 


B. Some Interesting Properties of Poisson Process 

We consider here some interesting properties which are of great use 
in the study of various phenomena. 

We have shown that the interval X; ( — ti+ı — fj) between two succes- 
sive occurrences Е, Ej, (i > 1) of a Poisson process with parameter A 
has an exponential distribution with mean 1/A. Further, the following 
result holds. 

I. For a Poisson process with parameter A, the interval of time X 
upto the first occurrence also follows an exponential distribution with 
mean 1/A. For, 

Pr (X, > x) = Pr {N(x) 20) =e, (x > 0) 
In other words, the relation (2.1) does not depend on | nor on ti. 

The' same argument also leads to another very important property. 

Suppose that the interval X is measured from an arbitrary instant of time 
tı + Y (ү arbitrary) in the interval (4), ti) and not just the instant t; of 
the occurrence of £;, and Y is the interval upto-the occurrence of £j, 
measured from t; i- y, i.e. 

Y — ti, — (ti ү); 
Y iscalled random modification of X or residual time ot X. It follows that: 

IL If X is exponential then its random modification Y h 
nential distribution with the same mean. 

We may state the result also as follows. 

ПІ. For a Poisson process with parameter A, the interval upto the 
occurrence of the next event measured from any start of time (not neces- 
sarily from the instant of the previous occurrence) is independent of the 
elapsed time (since the previous instant of occurrence) and is a random 
variable having exponential distribution with mean 1/A. 

The implication of the above would be clear when viewed in the light of 
an illustrative example (see, Karlin, p. 176) of fishing (incidentally, poisson 
is the French word for fish) as a Poisson process (named after the cele- 
brated French mathematician Denis Poisson (1781-1840)). Suppose that 
the random variable №2) denotes the number of fish caught by an angler 


as also expo- 
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in [0, г]. Under certain “ideal conditions", such as (i) the number of fish 
available is very large, (ii) the angler stands in no better chance of catch- 
ing fish than others, and (iii) the number of fish likely to nibble at one 
particular instant is the same as at another instant, the process (N(f), 
1; 0} may be considered as а Poisson process. -The interval upto the 
first catch, as also the interval between two successive catches has the 
same exponential distribution: So also is the time interval upto the next 
catch (from an arbitrary instant y) which is independent of the elapsed 
time since the last catch to that instant y. The long time spent since the 
last catch gives “по premium for waiting" sofar as the next catch is 


concerned. 


C. Geometric and Exponential Distributions 

The geometric and exponential distrioutions are unique in the sense 
that these are the only distributions with memoryless or no-aging 
property. In fact, there is some close connection betwcen these two distri- 


butions. 
We show below how one сап get the exponential distribution from 


the geomctric distribution. 
Suppose that the occurrence (or non-occurrence) of an event E may 


happen only at the discrete time points 3, 23, 33,. . ., each subinterval 
being of length 3, the probability of occurrence being p — A3 and that of 


non-occurrence being q — ] А. 
The number of failures № preceding the first success has a geometric 


distribution 
Pr{N=k}= фкр, k=0, 1,2..... 


Now the event that the first success occurs at trial number (k +4- 1) ог 
beyond has the probability 2 ФР = q*p/(1 — а) = q*, and therefore the 
probability that the time T required for the first success exceeds Кё is 
given by 

Pr(T> к) = Хар = gk =(1 — А). 
Now suppose that è — 0 as k - co , such that 5 = г remains fixed, then 
as k — 00, 
Pr (T > t) = (1 — МЈ) = (1 — МЮ р > (e pt e7 
Thus T has an exponential distribution with mean 


d 
D 


Bia 


Example 2(c). Poisson process and geometric distribution. Consider 
two independent series of events / and F occurring in accordance with 
Poisson processes with mean af and bf respectively. The number M of 
occurrences of E between two successive occurrences of F has a geometric 


distribution. 
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Proof: The interval between two successive occurrences of F has the 
density f(x) = be-^* and therefore Рг (N =k), the probability that k 
occurrences of E take place during an arbitrary interval (between two 
successive occurrences of F) is given by 


Pr == E on 


0 


Len et beidt 
0 


‚© 
= "e [Fg (a8) dt 
о 


_ . bat 
~ (at bn 


b a yet. x 
-hirap AREE N S (2.3) 


Poisson Count Process: Let E and E' be two random sequences of 
events occurring at instants (/,,/5,...) and (A',f,...) respectively. The 
number №, of occurrences of E' during the interval (f, ,, tn) of length 
fj — tn- is called the count process of E' in E. If E is Poisson then such 
a process is called Poisson count process. We have proved above that if 
Е' is also Poisson, then the count process N, has geometric distribution. 
It can also be shown that N, (л = 1,2,...) are independently and 
identically distributed with a common geometric distribution. For more 
general results and for applications of Poisson count process in Ou«ueing 
theory, see Kingman (1963). 


Example 2(d). Suppose that and F occur independently and in accordance 
with Poisson processes with parameters a and b respectively. 

The time between every two occurrences of F is the sum of two 
independent exponential distributions and thus has the density 


f(x) = b?xe-^*, 


The probability that k occurrences of E take place between every 
second occurrence of F is given by 


[ema = ad еа), 


0 0 
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а RUEZ 
k! (a+ b+ 


= (k + 1) (b/(a + b? (аа +B) 


ПЕТ р NaN g 
HPP E E enia. 


The distribution is negative binomial. This also follows from the fact 
that we have here a sum of two independent geometric distributions; we 
have two successive intervals (of occurrences of F) in which the number 
of occurrences of E follows a geometric distribution (see Example 1(g), 


Ch. 1). 


4.3 Generalisations of Poisson Process 

There are several directions in which the Poisson process (which may 
be referred to also as the classical Poisson process) discussed in the previous 
section can be generalised. We consider some of them here. 


A. Poisson Process in Higher Dimensions 

We have considered so far the one-dimensional case: the occurrences 
take place at random instants of time 7 (say, 4, ta.) and thus we were 
concerned with distribution of points ona line. Instead, we may have the 
two-dimensional case. 

Consider the two-dimensional case, such that for the number (Aq) 
of occurrences in an element of area A a, we have, for infinitesimal A a, 


Pr (N(Aa) = 1) —AAa + о(Аа). 
Pr (Мл а) =k} =0(Aa), k 22 
and Pr {МД а) = 0} = 1 -- à Aa + o(Aa). (3.1) 


Thus, if the number of occurrences in non-overlapping arcas are 
mutually independent, the number N(a) of occurrence? inan area a will 
be a Poisson process with mean Ла. Here in place of one-dimensional 
t, we consider two-dimensional a. Similarly, we can describe Poisson 
process in higher dimensions. 


B. Poisson Cluster Process (Compound Poisson Process) 
We considered in Sec. 4.1 that only one event can occur at an instant. 
Now let us suppose that several events can happen simultaneously at an 


instant, Le. we have a cluster (of occurrences) at a point. We assume 


that: 
(i) The number A(/) of clusters in time f, ie.the points at which a 
cluster occurs constitute a Poisson process with mean rate A. 

‚(Чу Each cluster has a random number of occurrences, i.e. the number 
X, of occurrences in ith cluster is a.r.v. The various numbers 
of occurrences in the different clusters аге mutually independent 
and follow the same probability distribution: 


8(45-95/1982) 
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Pr (X; =k} = py, k — 1, 2, 3, ... (3.2) 
i=, i . 
having p.g.f. 


P(s)= È рь. (3.3) 
k=1 


Theorem 4.6. 1f M(t) denote the total number of occurrences in an 
interval of length ¢ under the conditions (i) and (ii) stated above, then the 
generating function of M(t) is given by 


G(P(s)) = exp [At {P(s) — 19]. (3.4) 


Proof : M(t) isthe sum of a random number of terms, i.e. 
NG) 
мау= X X, 
i=l 


where N(t) is a Poisson process with mean At. 
Now P(s) is the p.g.f. of X, and G(s) is the p.g.f. of N(r). Thus, 


G(s) =exp {At(s — 1)}. 
Hence by Theorem 1.3.,the p.g.f. of M(t) is given by 
G(P(s)) = exp {At (P(s)— D). A 


Note: M(t) is also called a compound Poisson process. It isto be noted 
that M(t) is not necessarily Poisson. Poisson cluster processes (compound 
Poisson processes) arise in bulk queues, where customers arrive or are 
served in groups. In fact, a large variety of practical problems can be 
reduced to compound Poisson process (Feller II, p. 179). 

An important application arises in ‘Collective risk theory’. Suppose 
that claims against a company occur in accordance with a Poisson process 
with mean At, and that individual claims X; are i.i.d. with distribution 
{px}, then M(t) represents the total claims at epoch г. If A represents 
initial reserve and c the rate of increase of the reserves in the absence of 
claims, then the total reserve at epoch t is А + ct — M(t), and negative 
reserve implies ‘ruin’. 


Corollary: We have 
E (M(t)) = № E(X) 
and var {M(t)} = Xt E(X?)). (3.5) 
It can be shown that 
cov (M(t), M(s)) = (min (s, t)} {AE(X;9}. 


Compound Poisson process and linear combination of independent Poisson 
processes: 


Consider Example 1(f). 
The process X(t) =aN,(t)+ bN,(t) is a linear combination of two 
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independent Poisson processes. The process can also be expressed as 
the compound Poisson process 


ма) 
Хау= ХХ, 
i= 


where X;, the amount of commission received from a subscription, 1s a 
random variable such that Pr (X; = a} = § and Pr (X; = b) = }. 
If follows that 


E (X(t)) = E (N(0) E {X} = (60) -3-(2a -+ b) = 4at + 2bt 
var {X(t)} = 6t E {XP} = 61.3. (2a? + b?) = 4a?t + 2b*t. 


The two approaches are equivalent. The result may be stated in a more 
general form as follows: 

Геѓар> 0, k—1,2, ...,r(22)and Pr {Х = а} = рь for each i, 
Ўр, = \. Уф 

t 
Then X(t) = ey X; where {N(t)} is a Poisson process with parameter A, 
is a compound Poisson process. For t > 0, let {N,(t)} be the number of 
jumps of value a; for the process {X(t)} which occur prior to г. Then we 
have 
X(t) =a N (t) + + ar №), 

where {Nx(t), 1220) is a Poisson process with parameter Арк and N,(1),...5 
N,(t) are mutually independent. N(t) is decomposed into r independent 
Poisson processes Ng(t), k = 1, 2, ..., r. 


Example 3(a). Customers arrive at a store in groups consisting of 1 or 2 
individuals with equal probability and the arrival of groups is in accord- 
e with a Poisson process with mean rate À. 


anc 

Here рк = Рг (X; = K) =} (for k = 1, 2) 
— 0 (otherwise); 

hence 


P(s)= 2 режет 


and so the generating function of M(t), the total number of customers 
arriving in time / (by 3.4) is 

G(s) = exp {At B(s +- 5?) — 1]}. 
п number of customers arriving in time f is А. For A=} 


The mea 
— 4 minutes, the generating function will be 


per minute and ¢ 
exp [2 (8 G + 5) — 1}] = (exp (— 2) {exp (s + 55), 
y that the total number of arrivals is exactly 4 is 


fy uate wed 
e (tartar) = 0-14! 


and the probabilit 
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Example 3(b). Suppose that the number of arrival epochs in an interval 
of length t is given by a Poisson process (N(f)) with mean a and 
that the number of units arriving at an arrival epoch is given by a zero- 
truncated Poisson variate X;, i = 1, 2, ... with parameter A. Then the 
total number M(t) of units which arrive in an interval of length г is а 
Poisson cluster process with p.g.f. 


G(P(s)) = exp [at (Рз) — 1], 


where P(s) isthe p.g.f. of zero-truncated Poisson process (Exercise 1.5), 
namely, 
P(s) = (exp A— 1)? (exp (As) — 1). 


С(Р@) = exp a Gaat- 9 it} 


Hence 


C. Pure Birth Process : Yule-Fury Process 
In the classical Poisson process we assumethat the conditional probabi- 
lities are constant. Here, the probability that k events occur between 
t and г -+ h given that п events occurred by epoch / is given by 
ри) = Pr {N(h) = k | N(0) = n} =A h + o(h), К= 1 

= o(h), k22 

= 1 — Аһ -+- o(h), k = 0; 
р) is independent of n as well as t. We can generalise the process by 
considering that A is not a constant but is a function of n or ¢ or both; 
the resulting processes will still be Markovian in character. 


Here we consider that A is a function of n, the population size at the 
instant. We assume that 


ри!) = Pr {N(h) = k | N(t) = n} =A, h+o(h), k=1 
= o(h), k>2 (3.6) 
= 1 — ^h +- o(h), k —0. 
Then proceeding as in Sec. 4.1, we shall have the following equation 
corresponding to (1.10) 
Palt + h) = pat) (1 — Anh) + Pailt) An, h + olh), n 2 1. 
Proceeding as before, we get 


p, (t) = — pit) + M Pailt), n21 (3.7) 
y(t) = — Mp). (3.8) 
For given initial conditions, explicit expressions for p,(t) can be obtained 


from the above equations. 

We shall consider here only a particular case of interest, the case 
An = mA and describe a situation where this can happen. 

While the above process is called pure birth process,the process corres- 
ponding to A, = пл is called Yule-Furry process. 
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Yule-Furry Process: 

Consider a population whose members are either physical or biologi- 
cal entities. Suppose that members can give birth (by splitting or other- 
wise) to new members (who are exact replicas of themselves) but cannot 
die. We assume that in an interval of length Л each member has a 
probability A% -+ o(A) of giving birth to a new member. Then, if n 
individuals are present at time /, the probability that there will be one 
birth between ¢ and t + A is п Ah -+ o(h). If N(t) denotes the total number 
of members by epoch ¢ and p,(t) = Pr {М(ї) = п), then by putting A,=nA 
in (3.7) and (3.8), we obtain the following equations for pa(t): 


P(t) = — nA palt) + (n — DApa(), п>1 (3.9) 
po (t) = 0 (3.10) 
If the initial conditions are given, explicit expressions for p,(t) can be 


obtained. 

Suppose that the initial condition is p,(0) = 1, р(0) = 0 for i 5 1, i.e. 
the process started with only one member at time = 0. The solution 
can be obtained by the method of induction as follows : 


For п = 1, we have е 
ру (t) = — Apt) 
Pit) = 667"; 
and putting p,(0) = 1, we have c, = 1, i.e. p,(t)=e™. 


For n=2, we have 
py (t) = — 2^ p(t) + A p(t) 


or рә (0-- 2A p(t) = A p(t) =A e=. 
This linear equation has the integrating factor e?" and therefore 
e*Np,(t) = Jaen- dt = ем + PAR 

р+(0) = 0, we have c, = — 1 

р) = €? (eM — 1) = е-М(1.— e»). 


whose solution is 


since 


and 
Proceeding in this way, it can be shown that 

palt) = M1 —ey?, n2 1. (3.11) 
Solving (3. 10) and noting that (0) = 0, we get 

pot) = 0. 
The distribution is of the geometric form (see Exercise 1.2). Its p.g.f. 
is given by 9 
P(s, t) => (e(l р eMyay 5" 

п=1 


se^ 
=т=т (3.12) 


The mean of the process is given by 
вм) = ё; 
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Further, var(N(r))— e" (еМ — 1). 
D. Time-dependent Poisson Processes 
Here we assume that А is a non-random function of time г, and that 
Pr {М(Л) = k | N(t) = п} = ACOA + ah) ,k=1 
= о(Л) Yum (3.13) 
— 1 —X(0h -i- o(h) , k = 0; 
then we geta time-dependent Poisson process. Here the postulate II 
of the classical Poisson process regarding homogeneity does not hold good 


and this process is called a non-homogeneous Poisson process. The 
equations of the process which can be obtained from those of Poisson 


process by replacing А by А (t) are 


Рп (0) = —A(t)palt) + Np) nD 1 (3.14) 
po (t) = Xp). (3.15) 
Theorem 4.7. The p.g.f. of a non-homogeneous process (N(t)) is given by 
Q(s, t) = exp {m(t)(s — 1)), (3.16) 
t 
where m(t) — Гоа is the expectation of M(t). (3.17) 


о 
Proof: Now Q(s, t) = E (sN?) with Q(s, 0) = E (sN() — 1. 
Hence Qs, E- 5) eu E gf 
= E(sNU*BD-NQO) уму 
= E {sNU+h)-Nityy EfsNO} 
= E(sNGH)-NU) Q(s, t), 


since M(t) and N(t + h) — N(t) are independent (by postulate I of Sec. . 
4.] which still holds good; only postulate II does not). 


Now E(sNucM)-NQY = E(sNU)) 
' = AB {N(A) = k}sk 
= {1 — X(0A + o(h)) + {А (2)А + о(В)} s+o(h) 
=1+(s— 1) AA + o(h). 
Hence we have 
Os, t h) = {1 + (s — IAA + o(4)} Q (s, t) 
Distt) — Ou. 


or 


6—12(09 (з) + 90. 


We have, taking limits of both sides, as й > 0, 
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д 
0%, N = 6— DAC) Qs, 0. 


Integrating we get 
Os, t) = C exp {(s — nf» (x)dx}. 
0 


Аѕ Q(s, 0) = 0, we have C = 1, so that 
Q(s, t) = exp {m(t)(s — 1)), 
t 


where m(t) = f A(x)dx. 


0 


Now E(N(0) = 


20(5. t) 


so that m(t) is the expectation of N(/). A 


Corollary: The probability of no occurrence in an interval of length t is 
given by 


p) = Qs, ї)| = exp { — m(t)} = exp { -[ A(x) dx). (3.18) 


The probability of exactly k occurrences in an interval (0, £) is given by 


1 6k 1 ПИ: 
Put) = т Q(s £) sao = ру (nec mm 


t t 
I 
= Rit [лода exp (— [оф k>1. (3.19) 
0 0 
That is, if we introduce m(t) as operational time instead of absolute time 
t, the probability px of exactly k occurrences in the operational time interval 
" k ur 
[0, т(ї)] is given by the Poisson expression [OP exp (ant). It can 
easily be verified that when A(t) = A, these lead to the corresponding pro- 
babilities of homogeneous Poisson process. 
Further generalisation can be obtained by considering А as a function 
of both папа f£, i.e. à = A«(t). кз 
The Polya process, which is such a generalisation, corresponds to 


1+ an 
м0) = 1+ at 


E. Random Variation of the Parameter a | 
In order to allow for lack of homogeneity and the influence of random 


factors оп the development of the process, some random variation is 
introduced in the parameter А of a Poisson process N(t). These аге 


considered below (see Cramér, 1969). 


+ 


120 STOCHASTIC PRUCESSES 


Case I : Suppose that A is а random variable with probability density 
function f(A), OS AX co. | | . 
The probability of n occurrences in an interval of length? is then given by 


o2 


My o. 
ра) = PrANG) = 0} = |: И етм fiayda. 


0 


The particular case when A follows a gamma distribution is rather well 
known. Suppose that 


TAA 3o- Id 
TU 
where k is a constant parameter. Then 


ЈА) = 


Prit) -(* а Pj pkg? , p=h\(t+h=1—q,n=0,1,. 


and N(/) follows a negative binomial distribution with index k. 

Case И Suppose that A varies ina random way as time proceeds. Here, 
we consider А to be a random function of time t, i.e. A = A(t) is a stocha- 
stic process. A special case of Case II arises where the process A(t) is 
Stationary. This has been studied as conditional Poisson process of 
‘doubly stochastic Poisson process’. 

A conditional Poisson process is essentially а non-homogeneous 
Poisson process whose intensity function A(t) (or mean value function) 
is a stochastic process. These processes with strictly Stationary intensity 
processes, have been studied by several authors (Bartlett, Kingman, 
McFadden, Serfozo cte., see Grandell (1976) for a detailed account). 

F. Renewal Process 

In the classical Poisson process, the intervals between successive 
occurrences are independently and identically distributed with a negative 
exponential distribution, Suppose that there is a sequence of events E 
such that the intervals between successive occurrences of E are distributed 
independently and identically but have a distribution not necessarily 
negative cxponential; we have then a certain ‚ generalisation of the 


classical Poisson process: the corresponding process is called a renewa 
process (for details, see Ch. 6). 


4.4 Birth and Death Process 


In Sec. 4.3 C, we considered a pure birth process where 


Pr {Number of births between ¢ and t + Л is k, 
given that the number of individuals at epoch / is п} 
is given by 
Anh + o(h), k=1 
plk, h | n, t) = J olh), k>2 (4.1) 
1 —A,h -+ o(h), k=0. 
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The above holds for all л> 0; 4, may or may not be equal to zero. 
Here К isa non-negative integer which implies that there can only be an 
increase by К, i.e. only births are considered.possible. Now we suppose 
that there could also be a decrease by k, i.e. death (s) is also considered 
possible. In this case we shall further assume that 


Pr (Number of deaths between t and 1 + / іѕ k, 
given that the number of individuals at epoch £ is л} 


is given by 
unit + 0 (h), k= 1 
q(k, h| n t) = fen, k>2 (4.2) 
1 — рй + о(Л), k=0. 


The above holds for п> К; further шу == 0. With (4.1) and (4.2) 
we have, what is known as a birth and death process. Through a birth 
we have an increase by one and through a death we have a decrease by 
one in the number of "individuals". The probability of more than one 
birth or more than one death in an interval of length / is o(//. Let N(r) 
denote the total number of individuals at epoch z starting from ¢ = 0 and 
let pa(t) = Pr (N(t) = п). Consider the interval between 0 and ¢ +h; 
suppose that it is split into two periods (0,7) and (f,¢+h). 
The event (N(t + А) = п, n > 1), (having probability pa(t : /)) can occur 
in a number of mutually exclusive ways. 

These would include events involving more than onc birth and/or 
more than one death between f and ¢+A. By our assumption, the 
probability of such an event is o(//). There will remain four other events 


to be considered: 


Ayj:(n — i--j) individuals by epoch ¢, i birth and j death between / 
and t+ №, i, j = 0, 1. 


We have 
Pr (Ago) = palt) {1 — Anh + о(Л)} {1 — рай + o(1) 
= p(t) {1 — An + шл) =- o(hyj; 
Pr (Ayo) = Pn—s(t) (Anat + 0005 {1 — pni A -+ oU) 
= pna) Any А Olh); 
Pr (Agi) = Ран) {1 — Алы h+ 0005 {ener A 000] 
= ран) Una h + о(й); 
Pr (А) = Palt) n + 0005 {ten 0 + 000] 
= (A). 
Hence we have, for n 2 | 
Pat + h) = p) {1 — Qu + e) hy + Pult) А-у 
+ Parat) Bar A+ 001) (4.3) 
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DiA(t + Л) — P(t) zd 
or hi 


— (An + Bn) Palt) + a4 Pa-y(t) 


o(h) 
-H рача Pra) + “at 


and taking limits, as л — 0, we have 
p; (t) = — Qa - qa) Palt) + Any paa (t) + Und Pn (t), nz1 (4.4) 
For n = 0, we have 


Polt +A) = pot) {1 — МЛ + 0000) + p(t) {1 — Xt + o(h)} {ugh + on) 


(4.5) 

= pot) — М pot) + ult py) 

or MM pt) = — Ay pit) + qa p (t) + m 
i Д 

whence we get 

D, (0) = – 2 Ра) + By рц). (4.6) 
If at time ¢ = 0, there were i (> 0) individuals, then the initial condition is 

Pr(O) = 0, ni; p(0) = 1. (4.7) 


The equations (4.4) and (4.6) are the equations of the birth and 
death process. The birth and death processes play an important role in 
queueing theory. They also have interesting applications in diverse 
other fields such as economics, biology, ecology, reliability theory etc. . 

Note: The result about existence of solutions of (4.4) and (4.6) is 
stated below without proof. 

For arbitrary ^4 > 0, y, > 0, there always exists a solution pa(t) (> 0) 


such that Z pa) < 1. If An, #n are bounded, the solution is unique and 
satisfies X р„({):- 1. 


Birth and Death Rates 
Some particular values of’, and y, 
When 24553, їе, 
the increase 
immigration. 


When àn = л, we have the case of (linear) birth; АЛ = nii may be 


considered as the probability of one birth in an interval of length / 
given that ; individuals are present (at the instant from which the interval 
commences), the probability of one individual giving a birth being M 
(ie. rate of birth in unit internal is à per individual). Here A = 0. 

When y, = u, the decrease may be thought of as due to a factor such 
as emigration. 


When pn — ni, we have the case of (linear) death, the rate of death 
in unit interval being p per individual. 


are of special interest. 
^, is independent of the population size n, then 
may be thought of as due to an external source such as 
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Particular Cases 


I. Immigration-Emigration Process 

For А, =A and р, = we have what is known as immigration- 
emigration process. The process associated with the simple qucueing 
model M/M/1 (which we discuss in Sec. 10.2) is such a process. 


11. Linear Growth Process 

(a) Generating function : In the Yule-Furry process (Sec. 4.3 B) 
one is concerned with a population whose members can give birth only 
(to new members) but cannot die. Let us consider the case where both 
births and deaths can occur. Suppose that the probability that a member 
gives birth to a new member in a small interval of length / is АЛ -+ o(h) 
and the probability that a member dies is wh + 0(/). Then, if n members 
are present at the instant f, the probability of one birth between г and 
t+ his nA h+ o(h) and that of one death is nu h + o(h), п> 1. 

We have thus a birth and death process with 


An = MA, pn = np (nz 1), Ag = Yo = 0 


If X(t) denotes the total number of members at time z, then from (4.4) 
and (4.6) we have the following differential-difference equations for 


Patt) = Pr {X(t) = ny 
p. (t) = — nA + шр!) i A — Dpsa(t) i e(n- pat) nal (4.8) 
Po (1) = 010). (4.9) 
If the initial population size is i, ie. X(0) =i, then we have the initial 
condition p(0) = 1 and p,(0) = 0, n Ai. 
Let 
P(s,t) = E p(t)” be the p.g.f. of { pa(t)}. 
n=0 


Then 


oP n-l OP S US 
P Esp? and T= S руу". 


Multiplying (4.8) by s" and adding over л = 1, 2, 3... and adding (4.9) 
thereto, we get 


ve =—(A+ к) Харц +А E) AU — Dps-.(t)s" 


+ ТЕС + Ора" + рц } 


әр 
=- +022 OP уз 2Р Pas 


ар 
z(p—(Q us T3553 (4.10) 
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P(s, t) thus satisfies a partial differential equation of Lagrangian typc. 
We shall not discuss here the method of solution; the solution with the 
initial condition X(0) = i, is given by 


x LEGE — S) 00—05) РОИ 
Pen [1 ж =-= ==] 


— eg-O-p)n — —Ae-0-p0Y. Ti 
(1 — 9-00) — fu — pe Oe ae ail 


=> А fr EI реи — 1 — eowuns 

Explicit expression for p,(t) can be obtained from the above by expand- 
ing P(s, t) as a power series in s. 

(b) Mean population size: We can obtain the mean population 


size by differentiating P(s, t) partially with respect to s and putting s = 1. 


It can however be obtained directly from (4.8) and (4.9) without obtain- 
ing P(s, t) asfollows : 


Let E {Хи} = M(t) = х np,(t) 
п=1 
апа E (Y*0)) = Муг) = Ў пры). 
n=l 


Multiplying both sides of (4.8) by л and adding over for n = I2/3:115 
we have 


2 прі (2) = —(A+p) = парда) А X n(n — 1)pyy(t) 
n= Г n=] 


"= 


тр Х щи + pag. (4.12) 
Now 


© 
Y 


Uo Ор) = 5 (0 ера (9 + Ea Dp, qn 
= Ми) + Ми); 
toe Оры) $ + Pmt) È a рео) 


= LO = pi} — (M(t) — poy 
= Mt) — Ми); 
and b np(t) = M'(t). 
n=l 
Hence from (4.12) we get 


MU) -- — Q p) Mt) HALMA M(0J + e {Myt)— MO} 


=(A— ш) M(t). 


MARKOV PROCESSES WITH DISCRETE STATE SPACE 125 


The solution of the above differential equation (which M(t) satisfies) is 
easily found to be 
M(t) = Ceber, 


The initial condition gives M(0) = È np,(0) =i, whence C = M(0) = i. 
п=1 


We have therefore, 

M(t) = ice», (4.13) 
The second moment AM.(f) of X(t) can also be calculated in the same 
way. 

Limiting case : As t— co, the mean population size M() tends to 
0 for’ <p (birth rate smaller than death rate) or to œ for A > p (birth 
rate greater. than death rate) and to the constant value і when А = p. 

(c) Extinction probability: Since Ay = 0, 0 is an absorbing state, 
ie. once the population size reaches 0, it remains at 0 thereafter This 
is the interesting case of extinction of the population. We can determine 
the probability of extinction as follows : 

Suppose, for simplicity that X(0) — |, i.e. the process starts with only 
one member at time 0. Then from (4.11) we can write P(s, 1) as 


a—bs а 1— Ьа 


- P(s = с 7 e T= с 
where a= yu(1— Oe} 
and £A — роти. 


Pr {X(t) = 0) = po(t), the constant term іп the expansion of P(s, f) аѕ a 
power series in 5, is given by 

a PE EA 

v Keur aN (4.14) 
The probability that the population will eventually die out is given by 
lim pọ(t) as 1 > oo and can be obtained from the above by letting t — oo. 


If à > p then 
{ бө ес le era 
m Po") gus A— pene) 


If à <p, then 
р етеу, 


lim Pp ( ~ dim А рев d. (4.15) 


te Dr 
and lim p, (D^ 0 for nz: 0 
1» © 


In other words, the probability of ultimate extinction is 1 when p. > А (i.e. 


when the death rate is greater than the birth rate) and is »/A when u < А. 
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IH. Linear Growth with Immigration 

In IL we have А, == 0 and, as a result, if the population size reaches 
zero at any time, it remains at zero thereafter. Here 0 is an absorbing 
state. If we consider à, — т + a (a > 0), un = np (п > 0) we get what 
is known as a linear growth process with immigration, where 0 is not 
an absorbing state. 


IV. Immigration-Death Process 

If * 24, =A and p, = np, we get what is known as an immigration-death 
process. This corresponds to the Markovian queue with infinite number 
of channels, i.e. the queue M/M/oo. 


V. Pure Death Process 
Here A, = 0 for all л, i.e. an individual cannot give birth to a new indivi- 
dual and the probability of a death of an individual in (t, t -+ А) is 
uh i-oUr. Then, if n individuals are present at time t, the probability 
of one death in (z, t +- А) is ny + o(h). 

The birth and death process is a special type of continuous time Markov 
process with discrete state space 0, 1, 2, ... such that the probability 
of transition from state i to state jin At is o(At) whenever i=j > 2. 
In other words, changes take place through transitions only from a state 
to its immediate neighbouring state. 

In the next section we consider certain aspects of more general conti- 
nuous time Markov processes with discrete space. 


Such processes are 
also known as continuous time Markov chains. 


4.5 Markov Processes with Discrete State Space 
(Continuous Time Markov Chains) 


A. Introduction 


Let Y(t) Беа continuous parameter Markov 
NE ОО Seo) 
^ Further let X(t) be time-homogeneous or Stationary; then the (transi- 
tion) probability of a transition from state i to state j during the time 
interval from epoch Т to epoch Т -+ t does not depend on the initial 


time T but depends only on the elapsed time t and оп the initial and 
terminal states Тапа j. We can thus write 


Pr (X (T - t) (Туш i}= 


Process with state space 


Pi; (t), (5.1) 


| БУ ТОЙ, д) 1. fio; 
In particular, we have 


Pr (X (0 =j] X(0) = i} = р) (0). 
We have ' 


0 < pi; (f) < 1 for each 5g 
and Ур) @) = 1. 


/ 
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Let p; (t) = Pr (X (t) = j} be the state probability at epoch г; then 
pi (t) = Pr {X(N =j} 
арлы and X(0)— i) 


=EPr {X(0) =i} Pr {X (0 1X0) — 1) 
= 2 Pr {Ж (0) = ihip): (5.2) 
We һауе 2p;(t)==1 for each t > 0. 
J 


Let us denote the transition probability matrix by 


P (t) = (pi; (0). (5.3) 
Setting р; (0) = 8;;, we get 
P (0) = 1. Ў (5.4) 


We shall assume here that the functions p;; (f) are continuous and 
differentiable for t > 0. (For details, see Chung). 


The waiting time for a change of state: Suppose that X(t)is a homo- 
geneous Markov process and that at time f, = 0 the state of the process 
X (to) = X (0) =i is known. The time taken for a change of state from 
state i is a random variable, say t, which is called the waiting time for a 
change of state from state i. 

We have 

Рг {т > s+t| XO) =i} 

=Pr {r> s+t| XO =i, > s} Pr {r > s| XO) =i} 


=Pr {7 >s +t | X(s) = i} Pr {7 > s | X (0) =i}. 
If we denote F(t) = Pr {т > t |X (0) = i}, t > 0 then the above can be 


written as 
F(s + t) = F(s) F (f), for s, t > 0, 
which is satisfied iff F(t) is of the form 
F(t)=e™, t>0,4> 0. 
Thus the waiting time т has exponential distribution with parameter А, 
which is called the transition density from state i. 


B. Chapman-Kolmogorov Equations 
The transition probability p; (t + T) is the probability that given that 
0, it is state j at epoch ¢ + Т; but in passing 


the state was i at epoch 
from state i to state j in time (f + T) the process passes through some 


state k in time t£. Thus | 
pult + T) = ZPr (Xt + T) =} Ха) = k | X0) = i} 
ү k 
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— £Pr(X(0) — i, X(t) —k, X(t + T) = j}/Pr {X(0) — ij 
k 


Pr(X(0)— i, XO=K} ., Pr{X(0)—i, X(t)—k. Xt+T)=}} 
-Pri xO} ; Pr(X(0)—i, Xi) k} 


= ZPr{X(1) = k | X(0) = i} Pr {XC + T) = j| X(0) = i, X(t) = К}. 
k 


Since X(t) is a Markov process, 
Pr(X(t 1 Т) 2 j| X(0) — 7, X(t) = К} 
= Pr {X(t -+ T) = j| X0) = k} 
= pxi(T). 
Thus we get the relation 
pit +- T) = X pu(Dpi (Т), (5.5) 
which holds for all states i, j and t > 0, T > 0; it is called the Chapman- 
Kolmogorov equation. In matrix notation it can be written as 
P(t + T) = P(t)P(T): (5.6) 


The equations (5.5) and (5.6) correspond to (2.4) and (2.5) (of Ch. 3) 
respectively which hold for discrete space Markov chain. 
Denote the right-hand derivative at zero by 


1 » 
a = ду PifDeso: ij (5.7) 


d 
ai = gy Pil шо 


Then 
y РЕКА) — pi;(0) ОЛС) 
j— lim A ——— 
RE eT imt st 
or РҚА) = at 1- o(At), ij (5.8) 
im POM) — pi) у. pit) — 1 
2 1 ii - ii 
or Pit) — 1 + ае + o(At). (5.9) 
It can be seen from the above relations that 4; > 0, i Aj and aj < 0. 
From Z pi(t) = 1, using (5.7) we get 
Jj 
2 арс 0 
j 
or È aij = — aj. (5.10) 
iz " 


The quantities a;; are called transition densities and the matrix 
A — (aij) 
is called the transition density matrix or rate matrix of the process. The 
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matrix is such that 


(1) its off-diagonal elements are non-negative and the diagonal ele- 
ments are negative; 

(2) the sum of elements of each row is zero, the sum of the off-diagonal 
elements being equal in magnitude but opposite in sign to the 
diagonal elements. " 


Differentiating (5.5) with respects to T, we get 
T ERA = 101 н. а 
Py (t ЕТ) gp pit + T)s- 2 pat) тү РЕТ). 
Putting T := 0, we get 
p, (0) = 2 patas; (5.11) 
Or, in matrix notation 


Р'(г) = P(t)A. 
Similarly we can get (5. 11a) 


d 
dT РАТ) = Е акрку( T). 


Replacing Т by t, we can write this as 

pij (P) =È arpus 0) (5.12) 
or P'(t) == AP(t). (5.12a) 
Equations (5.11) and (5.12) which give Chapman- Kolmogorov equations 
as differential equations are called respectively Forward and Backward 
Kolmogorov equations. 


Solution of the Equations for a Finite State Process 

When the rate matrix is given, the equations (5.11) of (5. 11a) together 
with the initial conditions руу = 8; (or P(0) = Г) yield as solution the 
unknown probabilities p;;(t) We consider below a method of solution 
for a process with finite number of states. From (5. 11a) we see at once 
that the solutions can be written in the form 


P(t) = e“, (5.13) 
where the matrix 
= nmn 
At — | 
e HGT (5. 13a) 


eigenvalues of A are all distinct. Then from the 


Assume that the 
(see Sec. 1.7 or 3.4), we have 


spectral theorem of matrices 
A= HDH”, 


lar matrix (formed with the right eigenvectors of 
al matrix having for its diagonal elements the 
0 is an eigenvalue of A and if d; #0, i = 1,...,т 
then 


where H is a non-singu! 
A) and Dis the diagon 
eigenvalues of A. Now, 
are the other distinct eigenvalues, 


9(45-95/1982) 
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We then have 


-os 
a 
"s 


оо dp 
and А" = Нр"Н-1. 
Substituting іп (5.13), we get — ' 


© —1 
P()— 14. 5 Cmm 
нез п! 
= нр z ^ Ha 
n= n! 
= He? HA, (5.14) 
where 
1 0 0 
0 e^! 0 
et = А 
Е, 


The right-hand side of (5.14) gives explicit solution of the matrix P(t). 
Note that even in the general case when the eigenvalues of A are not 
necessarily distinct, a canonical representation of A= SZS-! exists 
(sec. 3.4). Using this, P(t) can be obtained in a modified form. 


Example 5(a). Poisson process: If events occur in accordance with 4 
Poisson process N(t) with mean At, then 


Pii (At) = Pr {the process goes to state ¿+ 1 from 
state i in time At) 


— Pr (one event occurs in time At} 

= Pr {N(At) = 1} 

-= At + o(At), 

P1,i(At) = 1 — Mt + o(At) 
and Di, (At) = o(At), j 5 1, i41. 
By comparing with (5.8) and (5.9), we have 
ала =À, dj = — 2, а, = 0 for j #i, 14-1. 

The Kolmogorov forward equations become 
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Pid) == = Ар) 
putt) = — Ар) Fama) j 7 i1, i4 2,. 
Let p,(t) = Pr {N(t) ^j) and p,(0) — 0. oe 
, ,p.(0)—0,nz^0. Usi 
we get p(t) = Pot), j~ 0, 1, 2.... Thus (5.15) become PAG 2) 
(1.11) and (1.12) obtained directly in terms of p;(7). cal with 


(5.15) 


Example 5(b). Two-state process: Suppose that a certain syst 

considered to be in two states: “Operating” and “Under 2 еш m be 
ted by 1 and 0 respectively). Suppose thatthe lengths of EU ( eno- 
and the period under repair are independent random unie pagoa 
negative exponential distributions with means 1/5 and l/a E aes 
(а, b 2 0). The evolution of the system can be described b; pode 
process with two states 0 and |. y a Markov 


Now, 
Po (At) == Pr (change of state from 0 to 1 in time Ar} 
= Pr {repair being completed in time Ar} 
= ал + o(At) 
and 


Py(At) = Pr {change of state from 1 to 0 in Ar} 
== bAt + o(At) 
Thus the transition densities are 
dy = а. аы == Б 
апа (dts mA Gia b 
so that у 


(5 2) 
The Kolmogorov forward equations, for i = 0, 1, are 
pit) = — ари) + b palt) 
palt) = ара) — b Palt). 


Now we proceed to find the transition probabilities p,;(t). 
Using 
Polt) + Palt) = 1 Prot) + Pu(t) = 1, 


we get 7 
polt) — (a+ b) Pot) =b 
pat) + (a + b) Pult) = 4. 


and 
st of these differential equations is 


The solution of the fir 
Poolt) im reas i Cet, 
a 


With P) = 1, we get C= acp 


, so that 
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a 


р eg (nbi. 
ат 


b 
Poo(t) = a+b a 


a a 
Hence Peat) = 1 — Pool!) = 7175 тту 


Similarly, the solution of the second differential equation with the initial 
condition p,,(0) = 1, gives 


е-(а+ 


b 
a+b 


and hence Pit) = 1 — put) 
b b 


a+b a+ 


en (atbyt 


a 
Palt) = a+b ks 


е-(а+би, 


Let p,(t) be the probability that the system is in state j at time f, j = 0, 1, 
and let po(0) = 1, р„(0) —0, n = 0. Then p,(t)=poj(t), j = 0, 1. 


Alternative method: This is considered to show how to proceed with 
the matrix or algebraic method of solution outlined above in case the 
Kolmogorov differential equations are not easily solvable. In the example 
under consideration, it isnot the case. However, the solution given below 
is expected to serve as an illustration of the method. 


In matrix notation, the Kolmogorov equations can be put as 
P(t) = (ру!) = P(t) А, 
whose solution (as given in 5.14) is 
P(t) = Her H=, (5.16) 
where А = HDH. 
The matrix A has two distinct eigenvalues 0 апа — (a+ Ь). 


The right eigenvectors corresponding to the eigenvalues 0 and — (a+), 


are the column vectors (1, 1)’ and (a, — Б) respectively. The matrix 
which is formed by these two eigenvectors is 


1 a 
н-(1 a) 
b/(ad-b)  а|(а+Ьу 
Ui 
юш (шү 1630) 
The diagonal matrix D formed with the eigenvalues as diagonal ele- 


ments is 
0 0 
2 - (5 aed 
1 0 
and ей = (0 "- E 


Substituting these values in (5.16), we get Pi;(t) as obtained above. 
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C. Limiting Distribution (Ergodici 
godicity of homogeneous Mark 
me iar the result on limiting distribution of certain Бр сг ae 
chains. heorem 3.6 states: For an irreducible ergodic Mark АЕА 
the limits асац 
lim р = 
п» ә Pik Yk 
exist and form a probability distribution 
Е j . Furth imi 
obtained as unique solutions of the equation ed E. 


WP — 1) —0. G.17) 
where V = (Vores =s Vin ..:), P = (р). 
A similar elegant result holds for continu 
ous рагатеќѓе! 
ses as well. We shall state the result without Sure ee рг осо 
definitions for the classification of the states (which constitut oy pee 
state-space) will be used. чела discrete 


Theorem 4.8. Suppose that the Markov is ti 

r В process {X(t 4 

ous, irreducible and ergodic; also that its t.p.m. У a E "n DE a 
1, 2,... and the matrix of transition densities (or rate matrix) is bse Ds 


А = (ay), 
where ар = P'y (t) li-o- 
Then given any state J, 
Jim pij (t) = v; (5.18) 
exists and is the same for all initial states ; —0, 1, 2, .... The asym 


ptotic values уу respresent а probability distribution, i.e. 
0xvj&€1, Хуу = 1. 
1 


values v; can then be determined as solutions of the system of linear 


The 
equations \ 
Хна =0 j=0,1,2,... (5.19) 
or in matrix notation, 
VA —0, V — (v Y». ) (5.192) 
by using the normalising condition 
У у; = 18 
j 


Note 1: The eq. (5- 19) сап be obtained from the forward Kolmogorov 
eq. (5.11) by putting 

lim pit) = vj and lim р;(1) = 0. 

m 249 
Note 2: Тһе eqs. (5.17) and (5.19а) for discrete and continuous 
parameter processes respectively are similar in structure. The matrices 
(Р --1) and 4 both have non-negative off-diagonal elements, strictly 
negative diagonal elements and zero row sums. If the number of states 
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are finite, say, m, then both (P — J) and A are of rank (m —1). Then V 
can be easily determined from any of the (m — 1) equations (out of т 
equations contained in the relations (5.17) cr(5.19a)) and the normalising 
condition X уу = 1. 

Jj 


Example 5 (c). Consider the two-state process given in Example (Р). 
Herc . 
vo = lim po(t) = lim pyo(t) = b/(a + b) 
1» = t>o 


and M jim Polt) = Jim Daft) = a/(a + 5). 


These limiting probabilities can also be obtained from the equations 
(5.19) which become 
—a%,+0%,=0 
ау – Б у, = 0. 


Here there аге two states and so the matrix A is of rank 1 and the two 
equations are equivalent. Each of them yields v, = (a/b) vo. Using the 
normalising condition v, -} v, = 1, we get 


va = b[(a + Б), v, = а/(а + Б). 


When the number of states is finite and when only the limiting probabi- 


lities v; are needed it is easier and more convenient to determine them 
from (5.19) or (5.19a). 


Example 5(d). Suppose that in a queueing system which has 
nels, the demand for service arises in accordance witha Pois 
parameter а. Suppose that the servicing time in each char 
tial with parameter b. Further suppose that there is no 
in the sense that a demand which arrives at the moment when any 
channel is free is received and is processed, whereas a demand which is 
received when all the m channels are busy is rejected and leaves the system. 
This is called a Joss system. 

Suppose that state k implies that k channels are busy. 
Markov process with (m- 1) states 0, 1, .. - Mm, 
babilities are given by 


m service chan- 
son process with 
inel is exponen- 
Storage facility, 


Then we get a 
The transition pro- 


Рыя+а (At) = Pr {one demand is received for processing in time A t) 
=аА1-+-о(Ат),0<<т 
Pij (А1) = о(Ат),}>1-+-1. 

Pr {one service demand is met in time At} 
=bAt+o(At), 

and so if i channels are working 

Pr {one service demand is met in time A t} 
=ibAt+o(At) 
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i.e. Pii (At) =i bAt о (At) 
and pi,j (At) =0 (At), jc i — 1. 
Thus do = а, dy = — a 
crat ,7=1—1 
ES sJ=itlel<i<m 
= — (a+ ib) ,j=i 
Am,m-1 = Mb, атт = — mb 
—a a 0 0m 
b —(a+6) Pel. em 
A=| 0 2b a zlar-pb) га 10 0 
The equations (5.19) become 
—avy+by,=0 
ау; — (a+ jovi + G + Dbra = 0, f= 1,2, ...,m—1 


аут-д — DMV m = 0. 
The solution of these equations can be obtained recursivel 
first cquation ely. From the 
v, = (a/b) vo. 
Writing the second equation with / = 1 and putting there this value of 
уу, we get 

v, = $ (a/b)? v. 

Proceeding in this way, we get 
Ашы (1//!) (alby Poppi 105: 152, ane m. 


m 


Using Ў 1; == 1, опе gels Yo == 1/( 2 (ui) (a/b)'). 
j20 d-0 | 


Formulas giving v; are called Erlang's formulas (see Sec. 10.4D). 
Example 5(©). Machine Interference Problem. Here there are m 
machines. Each of the machines operates independently and is serviced 
by A single servicing unit in case of break down. The operating time 
and servicing time of each machine are independently distributed as 
ibution with parameters b and a respectively. Then the 


exponential distrib D rs 
number of machines in operating condition at time f constitutes a Markov 


process (X(0). t20) with state space 0, 1, ..., m- 
( Incidentally, the rate matrix here is exactly the same as that of the 
preceding example; the limiting probabilities v; are given by the Erlang's 


loss formula.) 
nnel service system. Consider Example 5(d) with 


wo-cha 
e service channels are numbered I and П and 


Example 5(f). T 
that thi 


m:=2. Suppose 
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suppose that we are interested in whether particular channels are busy (B) 
m (F). Let the ordered pair (/, j) denote the state of the system, where 
i refers to that of the first channel and j to that of the second. The four 
states of the system (F, F), (F, B), (B, F) and (В, B) may be denoted by 
0, 1, 2, 3 respectively. The process (X(/)) denoting that the states of the 
Eon in terms of the two channels may be described by a Markov 
process with state space 0; 1, 2; 3. Moi 

Assume that when both the channelsare free a demand may join either 
of the channels for service with equal probability (3). Thus, when both 
the channels are free, demand to each of the channels flow in accordance 
with a Poisson process with parameter a/2. We have 


Po (A) = (4/2) At + о (М), j 142 


Py (At) = bAt +. 0 (At), j=0 
=at + o (At), j=3 

Po, (At) = bAt ~ о (At), ==) 
зам ~ 0 (At), p23 

Pa, (At) = bAt -- о (At), j=1,2 


and for all other combinations of i 5 j, Pi (åt) — о (At). 
Thus the rate matrix A will be 


—a al2 a/2 0 

b — (а -: b) 0 а 

b 0 -— (a -+ b) a 

0 b b = 2b 
The normal equations (5.19) become 

— йу, by, i by, 210, 
(4/2) w — (a + b) v, 3-0 
(a/2) vo — (a.3- b) v, -+ br = 0 
ауу + av —2by =0. 


From the second and third equations we get v, - v, and then from the 
first we get у, = (2b/a) v, and from the last уз = (a/b) уу. Utilising 
3 

У v, == 1, we at once get 
i=0 - 


‘ab 


"= "a= qi 3 ab 4- 2b 
2b? 

^o qa 2ab у эр? 
а? 


"= а Zab 4 2b* 


D. Graph Theoretic Approach for Determining V 


The graph theoretic approach for determining V in case of discrete para- 
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meter Markov chain (see Sec. 3.5) is also applicable to finite, irreducible 
Markov processes with continuous parameter. Incase of discrete Markov 
chains, arc ‘weights’ correspond to the transition probabilities p;;; in case 
of continuous Markov processes, the arc weights correspond to the transi- 
tion densities а; = (d/dr) pij{t)|--»- The same result (Theorem 3.7) leading 
to steady-state solutions can be applied to finite-state discrete parameter 
Markov chains as well as to finite-state continuous parameter Markov 
processes, provided these are irreducible (Solberg). This is a definite 
advantage of this graph theoretic approach. 
The procedure for continuous parameter process is illustrated below. 


Example 5(g). Erlang's formula: Consider the machine interference 
(or m-channel loss system) problem discussed above. 


Here for i =0, 1,2,....m—1,/#7 


а„ = а, j=it+l 
and for i = 1, 2, ..., m, j*1 
aj = ib, j-i—l 
and a, = —4, i=0 
= — mb, i-m 
= — (a +ib), 150, m. 
Also dij 220, jz5i— 1, E orb 4- T. 


The transition diagram may be shown as indicated below: 


a а 3 
" 


b 2b 


CX Xm) 


mb 


Fig. 4.3 


— sum of weights of all the intrees to the point 
г= 0, 1, 2,...,т. 


Denote с, 


Here there is only one intree to each point. We have 
co = (тё) (т — 1) b). . .(2b) (b) = m! b" 


c, = (mb) (m — 1) b)...(2b)a = т! ab™-1/1! 


c, = (mb) (т — 1) Б)...((г + 1) b) ar = т! arbn-rirt, 


so that Ў с=т! 5 (arb™=")/r!. Thus 
г=0 


r=0 
1 aY 5 4 (5) k=O i 23, 3.5 . 
„е J[ i5 5)” , т 
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4.6 Erlang Process 


A. Introduction | . А : 
Consider a series of events Е’ which occur in accordance with a Poisson 
process N(/) with mean a't. The random variable V; which represents the 
ac А А S АЖ 
interval between two successive occurrences has density h(x) E go 
having mean 1/2’. The interval separating k occurrences, i.c. the 
interval between the events numbered r and r ~ А (for k > 1) is given by 
the sum Sy -- X, --...-1 Хр of k independent exponential distributions 
each with mean 1/A'. Sy isa gamma variate (Theorem 1.10) with den- 
sity function 
AY 16-1 о 0 
= “И 2 
fA) Tk) a5 
=0 »t<0. (6.1) 
We have Е{$к} kiN and var (Sy) = АДА). 
Suppose that every Ath occurrence of £' is represented by а new series 
of events £, i.e. £ occurs for the first time at the kth occurrence of Е’, E 
occurs for the second time at the (2k)th occurrence of E’ and so оп. If 


X(t) is the number of occurrences of E in an interval of duration t, then 
we have 


Pr XQ) — nj — Pr (nk «z NU) < (n - 1) k— 1}, 


Q1) k-1 
= x Pr №0) =r} 


r—nk 


n>0 


(л+1)к—1 ему 
=. 6.2 
rank pl . (8:2) 
since N(t) is a Poisson process. 


Evidently, for positive integral k, 


a = WHEN RAT ом ору 
X Беру SS Gao 


n=0 n-Ü r-nk г! 


© о (At y 
AD a E 

Sun s! 
(X (t), t > 0j is a renewal counting process with probability distribution 
given by (6.2). The interval between two successive occurrences of X(t) 
which is the same аз the interval between А occurrences of N(t)) is a 
gamma variate with density given by (6.1). For k — 1, X(t) is identical 
with N (/) and is a Poisson process. 

(For thc p.g.f. of X(t) see Parzen p. 175.) 


B. Erlangian Distribution 
Putting 2 = AX, we find that the interval separating k events of E’ 


(which is the sum of & exponential variables cach with mean 1/К\) has 
the density 
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je eS 
poi 
Г(/) (6.3) 


H H ied g А á S 9 
(obtained by putting X = АХ in (5.1).) 
The distribution having tl bos 
rne g the above as its i n 
Erlangian or Erlang-k distribution (denoted. p funetton 1s called F- 
Ch. 1). y Ej) (see Example 3(f) 
If the- occureences of F ar | 
ш се аге such that the i 
successive occurrences of / are identically and Колы aed M 
with a common distribution Ej, then the number of ES ү пое 
T З 0) of F in an interval of length / constitutes what вл 2r 
rlang-k process. {X (t)} is a renewal process (Sec. 4.3F), Ven SA 
jo Be . е have 
(n-1)k&—1 (7EXM (fc 
Pr(X(0 = n} — > ош Ашу п> 0 
Eee Hoo З (6.4) 
(It may be said that а random selection 
ў preserves а Poiss 

(property 3 of Poisson process Sec. 4.1) whereas a debes 
10n 


destroys it. A systematic selection of Poisson process results i 
sın an. 


Erlang process.) 


Example 6(a). Suppose that a system fails at the cumulative elf 

shocks, If shocks occur in accordance with a boisson? TER 
parameter л", then the lifetime T of the system is a = офи: m 
being the sum of n independent exponential variates d d 
1/M. Hence Т іѕа gamma variate with its density given by un uo 


Example 6(b). Suppose that customers arrive at the desk of а receptioni 
in accordance with a Poisson process with a mean rate 2 (per Ki OE 
The receptionist immediately directs them alternately to FA z inute) 
A and 8 so that each counter gels every second customer. counters 

successive arrivals at А (or at В) is the sum 


‘The interval T between two 
of two independent exponential variates each with mean 4 (minute). T 
density function of T is obtained by putting k -- 2 anil 0а 6 "i 


We have 
(2r 107° = 
AD STO 02:0 
= 0. 1<0. 
E(T)= 2/2 — | and var (T) = 2/2? = 0.5 
and pr (T > 1 minute) - [ло dt — 367 — 0.406. 
1 


re is no arrival at B during the 


1) = 0} that the 
21,4’ —2, k — 2 and n -- 0 in 


The probability Pr {X( 
by putting /- 


first minute is obtained 
(6.2), i.e. 
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—2 (2) 
Pr(X(1) = 0) = ico — 0.406. 
Infact, | Pr(T > 1) = Pr (X(1) = 0). 


The probability Pr (X (1) = 1} of 1 arrival is obtained by putting 
alie 


€* QY _ 0.450. 
r; 


Pr {X(1) = 1) = > 


Example 6(c). Suppose that particles reach a nuclear particle counter in 
accordance with a Poisson process with a mean rate of 4 (per minute), 
but that the counter records only every third particle actually arriving. 

The probability that the number X (t) of particles recorded in time t 
is n is given by (putting k —3, »' = 4 іп (6.2)) 


3(п+1)-1 g-it (41), 
x LI, 
r=3n т. 
If t=1,n=1, then 


Pr (X (t) n) = п> 0. 


5 e 4r 
рг {Х(1)=1}= 2 < = 0.547. 
r=3 LI 


The interval T between two successive records (i.e. recorded particles) 
has the density function (putting a’ = 4, k = 3 in (6.1)), 


4312 с“ 
Де) = TO’ t>0 
=0, t<0, 
with E(T)=-3/4 and var (T) = 3/16. 


The probability that the interval T between two records is less than 
or equal to 3 (minute) is given by 


Pr(T«€4)-1— Pr(T» 3) 
—1—Pr(XQ) = 0), 


2 2 
=1— 2 $775. 933 
г=0 Г 
EXERCISES 


4.1 Divide the interval [0, /] into a large number п of small intervals 


of length Л and suppose that in each small interval, Bernoulli trials 
with probability of success АЛ are held (i.e. trials with only two 
outcomes, success with probability à% and failure with probabi- 
lity 1 — Ал). Show that the number ' of successes in an interval of 


length г is a Poisson process with mean At. State the assumptions 
you make. 


M. 
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4.2 


4.3 


4.5 


4.6 


4.7 


4.8 


If N,(t), N«(t) are two i 
A independent Poisson i 
meters A, A, respectively, then show that Vice а 


Pr (N (0 = k | Nn) + Мут) = n) -( x phar 


mi se Ж MO. EA). 9 = А0 +3) 
n a Poisson cluster process, su i 
аг , ose th 
distribution of the geometric tbe: eo sati 
pi — Pr (Xi = К) = дїр, 3.92 
Find the p.g.f., the mean and the i fat 
8-1., variance of Mi 
number of events). Show that the arrival stream AES total 
renewal process with inter-arrival time distribution шан 
A(t)=1—pe™ (t>0, 0<р< 1). 
The number of accidents in a town follows a Poi 
5 oisson pro i 
a mean of 2 per day and the number X; of people ОЕ А 
ith accident has the distribution шше 
1 
Pr (X, = К} = 3 (К > 1). 
Find the mean and the variance of the number of people involved 


in accidents per week. 
For the Pure birth process show that the interval Ty between the 


kth and (k + Dst birth has an exponential distribution with para 


meter Ax. 
Zipf's rank size law: Let Pr (asettlement S has п members at 
time г} = pa(t) and let Pr {a new member arrives in the settlement 


hat there are n at time t} =7 > 
that the settlement starts with 1 deni Toe, i. EM 
and that different types of individuals have différentrafes of Xm 
a, so that a is a random ‘variable with density g(a) = be-*5, a > 0, 
b > 0. Show that the probability distribution of S is given ‘by à 


in(t, t+ Л) given t 


fao g(a) da =rB (n, r + 1), n=1,4...5 


Pr {5 = nat time t}= 


Bis a Beta function and r = b/t. Show that for large n 


Pr {S > n |r} сп" 


where 


(J.K. Ord (1975)). 
ation of pure death process. If the process 


Find the differential equi 
starts with i individuals, find the mean and variance of the number 


N(t) present at time /. 
r the linear growth process, the second moment M,(t) 


Show that fo 
satisfies the d 
M'{t)= 


ifferential equation: 
2 = p) МИ) + A +) MD. 


Show that 


s 
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var (X(t)) = (IQ. + u)/G. — i) eey (oom — 1, AY 
—2iut,^ = y. 
i being the population size at г — 0. 
4.9 Find the probability of ultimate extinction in the case of the linear 
growth process starting with i individuals at time 0. 
4.10 Write down the differential-difference equations for the linear growth 
process with immigration Р 
An 7 MA -E 0, ра — пр. 
Show that 
M(t) = E{X(t)} 
satisfies the differential equation 
M'(?) —(A— y) M(t) i-a. 
Hence show that with the initial condition M(0) = i, if X(0) =i, then 
м) - — ge ri 1) + ber, эн 
and taking limit as à — p, show that 
M(t) = at -+ i for à = p. 
What is the limit of M(t) as t > co for à < u? 

4.11 Continuation of 4.10: Linear growth process with immigration. 
Given the initial conditions p,(0) = 1, Pr(0) = 0, n=40, show that 
the population size, after the population has been developing for 
time T, has negative binomial distribution with index ax, and 
mean 

afexp(A— ш) T—1) 
A-—u ` 
If ^ < y. the population size will have a steady state distribution. 
When ais small compared to à, show that, the population size, 
given that a population exists, follows a logarithmic series distri- 
bution. 
[Williamson and Bretherton, Ann. Math. Stat. 35 (1964) 284-297.] 

4.12 Forthe Yule-Fury process Starting with only one individual at t=O, · 
show that the density function of T, (the time at which the popula- 
tion first reaches the value n (7 1) is given by 

S(t) = (n — 1) e (1 — ern-e, t2 0. 
Find E(T,). [Note that T, <t = N(t) 2 n.] 
4.13 General form of Theorem 4.5: Fora Poisson process with para- 


meter A it is given that п occurrences take place in time f; show 
that the density function ofthe time of occurrence Zy upto the kth 
event (k — n) is given by 


А п! xk-1 xVr-k 
агат (1 3 s Ол? 
wu. xx 
Find the mean of Z,. 
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4.16 


4.7 


4.18 


4.19 


In Example 6(a), let n = 10, X = 1 

тео аел „АХ = I per hour; fi ili 
that the lifetime 7 (i) exceeds 10 hours, (ii) is is need 
tetas > ss than or equal 
If (М7), t > 0} is a Poisson process, show that the process 


X(t) = N(t + D — Nt) 


(where Lis a positive constant) is covariance stationa 

Consider a Yule-Furry process starting with a ae m b 
time ѓ = 0, and having birth rate à. Suppose that this fi = jx. 
(parent) is also subject to death, his lifetime being distribute "ps 
an exponential variable with parameter y. Find the diülributl of 
the number N of offspring due to this parent as well as his d = 
cendants at the time of death of the parent. к 
Fora count process N of E' in E, where E and E'arei 

Poisson processes with parameters à and A" Nl татна 
the joint distribution of № and TT being the interval betwee t 7 
successive intervals of E, is given by aii 


= ue Oe м)” 
fq, jee 1—0,1,2, 2. f 220; 


Hence find the (marginal) distribution of N as 


Pr(N—n)— [л t) dt, 
0 
and the distribution of Tas 
Дй= X fn. 
п=0 
Suppose that Nj(t), i = 1, 2 are two independent Poisson processes 
with parameters à; and X(t) = Ni( ) + N«(t) is their sum. Suppose 
that 7; and Z are the intervals upto the first occurrences of N(t) 
and X(t) respectively. Show that the random variable Z can be 
expressed in terms of T; as Z = min (Ту, Т,). What is the disribu- 


tion of Z? 

How do you general 
result hold if 7, 2 are 
rences of N;(t) and X(t) respectively? 

Write down the rate matrix for the birth and death process 
(Sec. 4.4). Assume that the limits Шш Pit) = уу exist. 

Obtain the difference equations involving v; from the Forward 
Kolmogorov equations. Using the method of induction, show 
that the solutions of these equations can be written as 


ise the above for i=1, 2, ..., n? Does the 
the intervals between two successive occur- 


ура = буза Vo 


Í N- 
I| =, fH 22 
г=1 Ur 7 


fine a probability distribution? 


where %=! and a; = 


When does {УЛ de 
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4.20 


4.21 
4.22 


4.23 
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Three-state process. Consider Example 54) with m= 2. Find 
pit). Obtain V from normal equation and also by using the graph 
theoretic technique. E . 
Obtain V for Example 5(f) by using the graph theoretic technique. 
Suppose that two machines working independently are looked after 
by one repairman. Suppose further that the break-downs of each 
of the machines occur in accordance with a Poisson process with 
parameter a, while the repair times of the two machines have 
independent exponential distributions with parameter b, and b» 
respectively. Suppose that W(R) represents the working (under 
repair) state of a machine and the ordered pair (i, j) (i. j= W, R) 
represents the state of the whole system so that the four, states of 
the system are 0: (W, W), 1: (W, R), 2: (R, W) and 3: (R, R). 
Determine the rate matrix. Evaluate V 

(i) from normal equations, 

(ii) by applying the graph theoretic technique. 

A repairable three-state process: The working of an automatic 
machine has been cited as an example of a three-state model by 
Proctor and Singh (1976). The states of an automatic machine on 
an assembly line could be good (state 0), (failed in mode 1 (state 1) 
and failed in mode 2 (state 2), (failure in mode | and 2 occurring 
due to carrying out some unwanted operations and due to ceasing 
to operate, respectively). If a; is the constant failure rate to failure 


mode i and b; the constant repair rate of failure mode i, then 
show that the rate matrix is 


— (a + a) a, а, 
A= b, —b, 0 |. 
b, 0 =b; 


If p(t) is the probability that the device is in state і at time /, 
i = 0, 1, 2, then show that the differential-difference equations of 


the system are 
Po (t) > — (а, F a3) рд) + by pi(t) + by p(t) 
pit) = apt) — bpd), i= 1, 2. 
If, at г = 0, the machine isin good state, i.e. p,(0) = 1, pi(0) = 0, 
i= 1, 2, then show that the solutions of the equations are (by the 
classical integration method or by applying Fourier transforms) 
pot) = 1— С, — С,— (A1 + Ay) e + (B, 4- B) есч 
Р) Ar er’ — Be, di 1,21 


where G= “3-1 A — Ubai — г) — “(bsi =3) 


їз КЁ r= rr — s) 
and r, s are the roots of 


x? — x (a, -+ ay + b, + by) + (bib, + a,b, + a,b) = 0. 
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Show that, in steady state 


a, bib, 3b, a. 
Po pss Р т 2 
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CHAPTER 5 


MARKOV PROCESSES WITH 
CONTINUOUS STATE SPACE 


5.1 Introduction: Brownian Motion 


Poisson process is a process in continuous time with a discrete state 
space. Here in a small interval of time Aż, there is either no change of 
state or there is only one change, the probability of more than one change 
being of the order of At. In this chapter we shall consider Markov 
processes such that in a small interval of time there is a small change of 
state or displacement. In such a process, changes of state are occurring 
continually all the time and the state space is continuous. An example 
ofa continuous parameter continuous state space Markov process is the 
Brownian motion process. The process derives its name from the 
English botanist Robert Brown, who first noticed in 1827 the erratic 
movement of particles suspended in a fluid under the random impact of 
neighbouring particles. Einstein was the first to put forward a satisfac- 
tory mathematical modelfor the physical phenomenon in 1905, while 
Wiener, in 1923, gave it a rigorous mathematical form. Considerable 
attention has since been given which has resulted in remarkable progress 
in the subject. It has now wide applications in such fields as economics, 
communication theory, management science etc. Because of the con- 
nection with the theory of diffusion, Markov processes with continuous 
state space are also known as diffusion processes. A particle under 
diffusion or undergoing Brownian motion is also known as a Brownian 
particle. We shall confine ourselves to one-dimentional Brownian motion, 
that is, to motion along a fixed axis. 

At epoch f, let X(t) be the displacement along a fixed axis of a particle 
undergoing Brownian motion with Х(0) = x,. Consider an interval 
(s, t) of time; let us regard this interval as the sum of a large number of 
small intervals. The total displacement X(t)— X(s) in this interval 
can be regarded as the limit of the sum of random displacements over the 
small intervals of time; these displacements may be interpreted as being 
caused by random walk performed by the Brownian particle over small 
intervals of time. Suppose that the random displacements are indepen- 
dently distributed. Then it can be seen that the central-limit theorem 
applies, whence it follows that the total displacement X(t) — X (s) is 
normally distributed. Further, suppose that the displacement X(t) — X (s) 
depends on the length of the interval (s, ¢) and not on the time-point s 
and that X(t) — X (s) has the same distribution as X (t -- h) — X (s + A) 
for any h > 0. 
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It is to be noted that here both time and space variables are conti- 
nuous. The equations of the process obtained by taking limits of both 
time and space variables will be partial differential equations in both 
time and space variables. These equations; called diffusion equations, 
will be discussed in Sec. 5.3. In Sec. 5.2 we develop Wiener process as 
the continuous limit of the simple random walk. 

It may be noted that there are some measure-theoretic subtleties 
involved in the passage from the discrete to the continuous case. Their 
considerations will, however, be beyond the scope of this book. 

We assume that the process X (t) is Markovian. Let the cumulative 


transition probability be 

P( з; x, t)= Pr(X() < xi K( = х), s < t (1.1) 
and let the transition probability density p be given by 

P(X» S ; х, t) dx = Pr {x < X(t) < x+ dx |X (s) = Xo}. (1.2) 


us process the transition probability depends only on 
e interval (7 — 5) and then the transition probability 
ers, Хо, X, t — 5. We denote 


For a homogeneo 


the length of th 
may be denoted in terms of the three paramet 


Pr {x < X(t--&) < x dx| X(t9 = x} by P (Xo x; t) dx" 
The Chapman-K olmogorov equation can be written (as the 


for any fy. 
н f (7.1) of Ch. 3) as follows: 


continuous state analogue o 


P(xy 83%) f) = Í d. P (xo 5; Z, у) P (Z, у; х, t). 
In terms of transition probabilities p (xo, 5; X, t), we have 


р (хо $; X, = [eco 5; 2, V) p (Z, у; х, t) dz. 


5.2 Wiener Process 


Consider that a (Brownian) particle performs a random walk such that 
in a small interval of time of duration At, the displacement of the particle 
to the right or to the left is also of small magnitude Ax, the total dis- 
placement X(t) of the particle in time £ being х. Suppose that Z; is the 
random variable which denotes the length of the ith step taken by the 
particle in a small interval of time At and that 

Pr {Z; = Ax} = р and Рг {2;= — Ах) =4р+9 = 1 
here р is independent of x and t. 

Suppose that the interval of length t is divided into n equal subintervals 
of length Atand that the displacements Z; i = 1,..., nin the л steps 
are mutually independent random variables. Then n (Ar) = t and the 
total displacement X (/) is the sum of n i-i.d. random variables Z; i.e. 


О<р< 1, м 


x()- X 2. 
а LE 
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dodi E {Z} =(р —4) Ах and var (Z;) = 4pq (Ах). 
Непсе Е{Х () = n E(Z) — t (p — 4) AxlAt, (2.1) 
and 


var X (t) = n var (Zi) = 4pqt (Ax)*/At. 
To get a meaningful result, as Ax — 0, At — 0, we must have 
e — a limit, (p — q) > a multiple of (Ax). (2.2) 


We may suppose, in particular, that in an interval of length t, X(t) has 
mean-value function equal to yt and variance function equal to o?t. In 
other words, we suppose that as Ax > 0, At — 0, in such a way that 
(2.2) are satisfied, and per unit time 


E(X(t) ^ u and var {X(t)}} > o°. (2.3) 
From (2.1) for t = 1 and (2.3) we have 
(шад Е. РОСА. аз 


А? AT : (2.4) 
The relations (2.2) and (2.4) will be satisfied when 

Ax = c (Аг)!, (2.5a) 

p — (1 + (Аг)! ?/в), q =4(1 — u (At)"?/0). (2.5b) 


Now since Z;are i.i.d. random variables, thesum È Z,=X (t) for large 


n is asymptotically normal with mean wt and variance o*t (by virtue of 
the central limit theorem for equal components). Note that here also Г 
represents the length of the interval of time during which the displace- 
ment, that takes place is equal to the increment X. (t) — X(0) We thus 
find that for 0 < s < t, X(t)— X (s) is normally distributed with mean 
& (1—5) and variance c*(t— s). Further, the increments X(s) — X(0) 
and X (t) — X (s) are mutually independent; this implies that X(t) is a 
Markov process. 


We may now define a Wiener process or a Browni 
as follows: 

The stochastic process (X(t), t > 0 
Wiener-Einstein process or a Brownia 
variance parameter с?, if: 

(i) X(t) has independent increments, i.e. for every pair of disjoint 
intervals of time (5, f) and (u, v), wheres < t < u < v, the random 
variables X (f) — X (s) and X(v) — X (u) are independent. 

(ii) Every increment X (r) — X (s) is normally distributed with mean 
e (t — s) and variance c? (t — s). 


Note that (i) implies that Wien 
independent increments and 
Since X (tr) — X (0) is nori 
o*t, the transition probabi 


an motion process 


) is called a Wiener process (or a 
n motion process) with drift и and 


ег process is a Markov process with 
(ii) implies that a Wiener process is Gaussian. 
mally distributed with mean vt and variance 
lity density function р of a Wiener process is 
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given by 
D(X, x; t) dx = Pr {x < X(t) < x + dx| X(0) = xo} 
ot (X 3r pt)? 
27057) epf 295] | ах. (2.6) 


A Wiener process X (t) with X (0) = 0, u = 0, с = 1 is called a standard 
Wiener process. 


5.3 Differential Equations for a Wiener Process 


Let X(t) be a Wiener process. We can consider the displacement in 
Such a process as being caused by the motion ofa particle undergoing 
displacements of small magnitude in a small interval of time. Suppose 
that (1 — At, t) is an infinitesimal interval of length At and that the particle 
makes in this interval a shift equal to Ax with probability p or a shift 
equal to — Ax with probability q— 1 — р. Suppose that p and q are 
independent of x and t. Let the transition probability that the particle 
has a displacement from x to x + Ax at epoch 5, given that it started 
from x, at time 0, be р(хо. x; 1) Ax. Further suppose that p(x, x; 1) 
admits of an expansion in Taylor's series, i.e. 

P(X, Xd Ax; t— At) = р(хо X; )—м c ax gn 


ась Axe Z2 + о(м). (3.1) 


From simple probability arguments,we have 
P (Xp X; t) Ax = p.p (Xo) x — ÂX; t— At) Ax 
+ q.p (Xo X + Ax; t — At) Ax. (3.2) 
Making use of (3.1), and cancelling out the factor Ax from both sides of 
(3.2) we get 


др _ 7) др 
р(хьхи) = P Qe xi — Atar АХ D ax 


д, 
+ 4(Ax)? а + e(At). 
Divide both sides by At. Using (2.4) and (2.5) and taking limits as 
At — 0, Ax — 0, we get $ 
^5 e 

д 0 л ag? Salty : D. i 

gi o ЁЗ A ы 2x Pv x; t) + 36 ox? р(Хо, X5 t). (3.3) 
ion in both the variables x and t, being 
d order in x. The equation is known 
f the Wiener process. One can likewise 
tion of the process in the form 


This is a partial differential equat 

Of first order in t and of the secon 

as the forward diffusion equation 0 

Obtain the backward diffusion equa 
2? 

5 8 р(хо, х; 0) + 39 ga Р(Х» 9; D. 3.4 

gu Р(х x3; 1) = ® ву Р(Х» ®› 0) + 40° gos PO ) (3.4) 


150 STOCHASTIC PROCESSES 


The solution of (3.3) as well as of (3.4) yields р(х, х; f) as a normal 
density of the form given in (2.6) (see also Sec. 5.5). It may, however, 
be easily verified that p(x, x; t) given by (2.6) satisfies (3.3) as well as 
(3.4). The equation for a Wiener process with drift u = 0 is known as 
the heat equation. 


Note: The partial differential equation (3.3) [(3.4)] is known as the 
forward [backward] equation because it involves differentiation in x[x,]. 
The reason why it is called diffusion equation is given in the next 
section. 

Tt is to be noted that in sec. 5.2 and 5.3 we have made the following 
assumptions: 

(i) inasmall interval of time Af, the displacement Ax is small 
(and that Ax = 0 ((AD9); 


(ii) E {X(t)} > ut in the limit; 
(iii) var {X(t)}}—> ої іп the limit. 
The quantity џ in (ii) may also be interpreted as 
LL E{X(t- At) — X(0) 
a fc wed (3.5) 
This implies that the infinitesimal mean (i.e. mean over Af) of the expec- 


tation of the increment in X(t) exists and is equal to a finite quantity y. 
The quantity o? in (iii) may also be interpreted as 


дхи м) ХШ. 

е ааъ 0.6 
This implies that the infinitesimal mean (mean over At) of the variance 
of the increment in X(t) exists and is equal to a finite quantity o?. 

For a Wiener process, џ and o? are assumed to be constants independent 
of t or of x (where X(t) = x). By considering the transition mechanism 
with v. and о? as functions of ог of x or both г and x, we get more 
general processes for which the equations corresponding to (3.3) and 
(3.4) will also be more general. We discuss below such tution 


5.4 Kolmogorov Equations 


Let {X(t)} be a Markov process in continuous ti + 

А 5 бте і 
state space. We make the following assumptions: For ip i DA 
@ Pr(] Xa) — XG) | 2 8| X0) x) o((— s), s ; 


Da bone ick chan during small intervals of time 
a tern а АДАТ) E 
2150 At 


— lim | (Y — х) р(х, t; y, t-: At) dy 


dtd 
| y-x іё 


eu, X). 
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In other words, the limit of the infinitesimal mean of the conditional 
expectation of the increment of X(t) exists and is equal to a(t, x), which 
is known as the drift coefficient. 
E([X(t + At) — XP | XO = х) 

At 


(iii) lim 
Atd 


= lim (y = x)? p(x, t; y, t + At) dy 
Ald 
ly-x| <8 
= bit, x). 


In other words, the limit of the infinitesimal mean of the variance of the 
increment of X(t) exists and is equal to S(t, x), which is known as the 
diffusion coefficient. 

A Markov process X(t) satisfying the above conditions is known as a 
diffusion process and the partial differential equation satisfied by its transi- 
tion p.d.f. is known as diffusion equation. We give below the equations 
(for derivation see Prabhu,’ Blanc-Lapierre and Fortet). 

Let X(t) be a Markov process satisfying (1), (ii) and (iii). If its transi- 
tion p.d.f. р(хо, £y; х, f) possesses continuous partial derivatives 

др д д> 


BP, 2 (дї, хр), руз 60 з)р), 


1) satisfies the forward Kolmogorov equation 
ар 123 j г> г 

Sie = (a(t, х)р) + toa (b(t, x) p). (4.1) 
This equation is also known as the Fokker-Planck equation. Suppose 
that p (Xy to; X 1) possesses continuous partial derivatives 


ap Op Op, 


9t Xo Ox, 


then p (x, t; x, t) also satisfies the backward Kolmogorov equation 


then р(хо, to; X; 


др a f op me д?р 
Ө — a(t Xo) ax, +5 (to, DET . (4.2) 
The diffusion equations for X(t) were first derived by Kolmogorov. 
Feller showed that under suitable restrictions the equations admit of a 
unique solution. Fortet established some very interesting and important 
Properties of the solutions. 


Particular case: If the process is homogeneous, then 
р (tgs ху; і, X) = р(х 33 1 — to), and a (t, x), b(t, x) 
are independent of /. ^ 
If the process is additive, ie. given that X (1) == xo, the increment 
X (t) — X (tj), depends only on to and ż (and not on xj), then 


р( X; b х) = P(X Хо; 1 D and a (t, x), b (t, x) 


are independent of x. 
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The Kolmogorov equations, in these cases, can be easily deduced from 
the general equations: 


5.5 First Passage Time Distribution for Wiener Process 


The possible realizations of a stochastic process are called sample paths 
or trajectories. The structure and the properties of the sample paths of 
a Brownian motion or Wiener process are the subject matter of deep 
study. Without entering into the subtleties (which are beyond the Scope 
of this work), we discuss here some results of Wiener process, using the 
property that the sample paths are continuous functions. We also make 
use of the simple but powerful ‘reflection principle’ (enunciated by D. 
André in 1887). The principle relates to the fact that there is a one-to- 
one correspondence between all paths from A (a, a) to B(b,, ba) which 
touch or cross the x-axis and all paths from 4’ (a,, — a,) to B (see, Feller 
I for details) We shall first consider the following from which the 
distribution of the first passage time will be derived. 


Distribution of the maximum of the Wiener process 


Lemma: Let X(t),0 < t « T bea Wiener process with X(0)—0 


and p —0. Let M(T) be the maximum of X(t) in Oc t« T, ie. 


M(T)= max X(t). Then for any a 0 
ostsT 


Pr(M(T) > a) = 2 Pr {X (T) > а). 
(This result was first obtained by Bachelier (1900).) 


Proof: Consider the collection of sample paths X (0,0 < t < Tsuch 
that X(T) >a. Now X¥(0)=0 and X(t) is continuous, there exists a 
time т at which X(t) first attains the value a(or X(t) hits first the value 
a). The time T, isitself a random variable. For: > Ta, X, (t) given 
below 

_ fX, tm. 
X«Q) Be Ха)» 1>T, 


gives the reflection of X(t) about the line x = a, Note that X,(T) < a, 
and that M(T)= max X()>a and M,(T)= max X.(t) > a; 
further, by symmetry the sample paths ¥ (t) and X.Q) have the same 
probability of occurrence. From reflection Principle, it follows that 
corresponding to every sample path X(t) for which X(T) > a, there 
exist two sample paths such that M(T) > a, Further, its converse is 
also true, viz., every sample path X (t) for which М(Т) >а corresponds 
to two sample paths ¥(t) with equal probability, one of the paths being 
such that X(T) > a, unless (X (T) = 4), whose Probability is zero. In 
fact, the set (M (T) 2 a) is theunion of three disjoint sets i 
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{М (Т) > a, X(t) > a), 
{M(T) > a, X(t) < a), 


{M(T) > a, X(t) == a). 
The probability of the third set is zero, while the'other two are mapped 
onto one another by refléction about the line x =a after the time Ta. 
Thus we have 
Pr (M(T) > a} = 2Pr (X(T) > a}. 
The above gives a heuristic proof of the lemma; as already indicated, 
a rigorous proof involves considerations beyond the scope of this book 


(see Karlin; Iosifescu and Таши). 
Let X (t) be a Wiener process with X(0)--0, »=0 and let 
M(t)= max X(s). Then from the lemma, we get, for t > 0, o = 1, 
Ossst 


Pr {M(t) > d) = 2 Pr (X (t) > a} 


- RI | exp (— 32/2) dy (5.1b) 
БЕД 


(by changing the variable to y — x/4/t) 
1 ave 
=O — 99/2) dy 
Fem J eec rm an 


= 2 {1 — Ф (a/v D}, (5.1c) 


® being the distribution function of the standard normal variate. 
By changing the variable to s = a*t/x*, (5. 1a) = be written as 
t 


" 2 
Pr {M (t) > а} = Fan Lr Тр 5) ds 520. (5.10) 
0 


Distribution of the first passage time to а fixed point 


Wecan use the lemma to obtain the distribution of the random variable 
Ta, the first passage time to a fixed point a (> 0) (or the time of hitting 
а fixed point а first), for a Wiener process Л(1) with X(0)- 0, 
u:=0. The time 7, for X(t) to hit the level a first will be less than t 
iff M(t). тах X(s) in that time is at least а. Thus for í > 0 


0.17217 


Pr{M(t) >а} Р {Т & 1. (5.2) 


STOCHASTIC PROCESSES 


154 
Е() = Js | exp (— x?/2t) dx (5.32) 
-—2(1— 9S (5.3b) 
= 788 je exp (- 5) ds, 52-0. (5-30) 


0 


The density function of Т„ is obtained by differentiating (5.3) with 
respect to t. Differentiation of (5.3c) (and also 5.3b) readily gives 


a 
frat) =F (= sigs rep (—5;), > 0. (5.4 


It may be easily verified that the Laplace transform f* (s) is 


O= | ena 
L] 
= exp (—a +/(2s)} (see Exercise 1.10 (viii)). 
It can be seen that no moment of Т, exists finitely. | 
Let us find the density function of M(t). The distribution function is 


G (a) — Pr (M (t) < aj 
=1— Pr (M(t) > а} 
= 1 — 2Pr {X(t) >а} 


2 
-1— VOD f exp (— x?/2t) dx (5.5a) 


a 
=1—2{1— 9 (d) (5.5b) 
= 20 (aj /t) — 1. 
Differentiating (5. 5а) or (5.5b) with respect to a, we get the density function 


gu (а) = G' f= Jom exp (- 5) а> 0. (5.6) 


The results given above (for с = 1) сап be suitably modified for any 
с> 0. 

Note 1: We have obtained the distribution of Ta by using the lemma 
which gives a relation between the distributions of the Wiener process 
X(t) and its maximum M(t). However, the distribution of T, can be 
obtained directly without bringing in the distribution of the maximum. 
In fact, it can be directly shown that, if X(0) = 0, a> 0, then 


Pr (T, < t) — 2Pr{X(t) > a). (5.7) 


For a proof of the above, see Prohorov and Rozanov (p.245), who use 
conditional expectations to obtain the above. The distribution of the 
maximum M(t) can then be obtained by using (5.2). 
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Note 2: Foran alternative approach to the distribution of Ta using 
differential equations, see Cox and Miller (pp. 210, 220, 230), who obtain 
the distribution for any p > 0 and o > 0. The density function /7,,(x) (for 
X(0) = 0, » > 0, с> 0) of Ta is found to be 


me (a= yxy 
Sra'X) o / (27 x?) pe | T i | r2 oe 


and its Laplace transform is 
f*(s) = exp [(a/c?) { + & — v (p° + 2597)]- 
The mean and the variance for Ta for џ 5 0 are given by 
E {Ta} = а[ and var (Ta) = a o*/p'. 

Note3: The function (5.8) with > 0 is the density function of the 
distribution of the first passage time of Brownian motion with a positive 
drift. This distribution having density function (5.8) is known as inverse 
Gaussian distribution because of the inverse relationship between the 
nerating function of this distribution and that of normal 

Such a distribution was also obtained by Wald as the 
stribution of the sample size in a sequential 
For properties of this distribution, see Johnson 
al applications, see Folks and Chhikara 


cumulant ge 
distribution. 
limiting form for the di 
probability ratio test. 
and Kotz (1970), and for statistic 
(1978). 
Example 5(a). Suppose that {X(t), 0 < t},is a Wiener process with 
X(0) = 0, and p = 0. Then 

Pr (X(t) < x) = Pr (lev t< xlo у 1) = ®Ф(х/з vt). 


Consider the process 
y(t) — X(1/t) in 0 « t <1 with Y(0) = 0. 


We have | 
E(Y(0) = 0 and var {¥(t)} = t? (o°/t) = c't. 


Further, 


pr (H(t) <a} = Pr € ХШ) < ») = Pr Us 2177} 
= Ф (yloy/t). 

Thus Y(t), 0< ! < 1 with Y(0) = 0 is also а Wiener process with p. = 0 

and variance c*t. 

a Wiener process X(t) with X(0) = 0. Its mean 


Example 5(b). Consider 
JU d variance function o*f. For 0«csct, the 


‚ value function is uf an 
covariance function is 
C(s, t) = cov (X5), X(0) = cov (Q9, XG) + X() — X) 
— cov (X(), X(s)} + cov (XG), X(t) — Х(5)} 


= cov {X(s); Х(5)), 
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since the process has independent increments. Thus 

C(s, t) = var {X(s)} = os 
(see also Exercise 2.7). The process is not convariance stationary even 
when p. = 0. 


Example 5(c). Suppose that X(t), t > Ois a Wiener process with X(0) = 0. 
Its first passage time, Ta, to a has the same distribution as l/u?, where u 
is a normal variate with mean 0 and s.d. c/a. For, the distribution func- 
tion of l/u? for t > 0, is 


F(t) = Pr (1ju* € t} = Pr {u®/(c/a)? > а/с? t) 
== Pr ((au/c) > (alas t)) + Pr {(аи[в) < — (alo 4/t)) 
== 1 — Ф(а!в yt) <- Ф(— alo yt) 
=2 (1— Ф(а/в 4/1), 


which is the distribution function of Ta (see equation (5.3b)). The distri- 
bution of 1/02 is inverse Gaussian. 


Example 5(d) Let X(t) be a Wiener process with p =0 and X(0) — 0. 
To find the distribution of Тш» for 0 < a < a +- b. 

Suppose fa is a value of Ta, i.e. X(t) reaches the level a for the first 
time at epoch fa. We may then consider that the process starts at (ta, a) 
and reaches the level (а + b), which is b units higher than а. Suppose 
that Tas — Ta is the duration of the interval at the ends of which X(t) 
first reaches the level a and then reaches first the level a-- b. Then T, 
and (То+ь — Ta) are independent random variables denoting first passage 
times to a and b respectively. The L.T. of the p.d.f. of T, is 


ha*(5) = exp (— a V 2)jo) 
and that of T; is - ES 
hy*(s) = exp (— b V/(2s)/o}. 
Thus the L.T. of the p.d.f. of Tax, is 


ha*(s) hy*(s) = exp (— (a + b) V(2s)/o} = he, (5). 


The distribution of Ta,» has the same form as that of T, and can be 
obtained by replacing a by (a + b) in the distribution of T,. 


5.6 Ornstein-Uhlenbeck Process 


We have seen that for a Wiener process X(t), the displacement Ax in a 
small interval of time At is also small, being of O(4/(At)). The velocity 
which is of 0(/(At)/At) = O(1/A/(At)) tends to infinity as At— 0. 
Thus the Wiener process does not provide a satisfactory model for 
Brownian motion for small values of t, although for moderate and 
large values of ¢ it does so. As alternative model which holds for small 


MARKOV PROCESSES WITH CONTINUOUS STATE SPACE 157 


t was proposed by Ornstein and Uhlenbeck in 1930. Here instead of the 
displacement X(t), the velocity U(t) == X'(1) at time t is considered. 
The equation of motion of a Brownian particle can be written as 
dU(t) = — 8 U(t)dt аба), (6.1) 
where — BU(t) represents the systematic part due to the resistance of the 
medium and dF(t) represents the random component. It is assumed that 
these two parts are independent and that F(t) is a Wiener process with 
drift р — 0 and variance parameter c?. The Markov process U(t) is such 
that in a small interval of time the change in U(t) is also small. Since 
F(t) is a Wiener process. we have from (6.1) 


lim E {U(t -- At) I: | U(r) — a} 


21-0 
ваіт ВАО) 
ТЕ] H 
— — фи, 
U(t +- Аг) — U(t = 
and lim уйт ) A (0) | UC) = 10 
4->0 
er (Ак тда: {AF(t)} 
lin cd AP 
= с?. 


In other words, the limits exist. So the process U(t) is a diffusion pro- 
cess and its transition p.d.f р(и, и, t) satisfies the forward Kolmogorov 
equation (4.1) with a(u, t) = — Ви and b(u, 1) = c°. That is, p satisfies 
the differential equation 

pid caque 

a P aD a (6.2) 
at U(0) =u and that as иу 4- o0, p> 0 and (8p[01)-» 0. 


the transition p.d.f. of U(t). It is more 
tion corresponding to (6.2) in terms of 


Let us assume th: 
The solution of (6.2) gives p» 
convenient to consider the equati 
the characteristic function of p, i.e. 

duo; 9, 1) =| ef p(uy; и, t) du. 


We have 


[ ge 2 (up) du — ei" w| — f 10 ei^" up du 
ди 


| eit p du 
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T „ёр др |] PIRE dp 
| hei | ie [e ay du 
=— ið {em p | — ið Í ею р du} 
=— 0%. 
The equation (6.2) then becomes 
2 2 


The equation (6.3) is of Lagrange type. It can be shown that 
(ug; 0, t) = exp (i0 ug e-&! — 3 025° (1 — e-8ty, (6.4) 


This is the characteristic function*of normal distribution with m(t) — 
u,e-*' and variance function c*(t) = o? (1 — e-?8048. 

In other words, the transition p.d.f. p is normal with mean value 
function m(t) and variance function c*(f) and p can be written as: 


Р; өл) = Tama P Go mOYBe()). (6:5) 


Thus the process U(t) is a Gaussian process with mean value function m(t) 
and variance function с“). U(t) is a Markov process but it does not 
possess independent increments like the Wiener process, U(t) is known 
as Ornstein-Uhlenbeck process (O-U. P.. For large t, m(t)— 0 and 
o°(t) > c?/28, i.e. the distribution of velocity is normal with mean 0 and 
variance o*/28, We thus get an equilibrium distribution and U(t) is:said 
to in statistical equilibrium. For small t, m(t) — uy and o(t)—> oF, 


Example 6(a). Joint distribution of U(t) and U(t 
equilibrium. For large г, the limiting distribution of U(t) is normal with 
mean 0 and variance c*/28 = o? (say). The conditional distribution of 
U (t+ х), given U(t)=u, is normal with mean ye-At and variance 
o°(t) = o?(1 — e~*F*)/28. Thus the unconditional distribution of U (t + т) 
has the following density 


+ т) when U(t) is in 


"= za | [2 (- 2з) ra 


exp {- zx) =x; ev] dx, 


= 1 x? 
= Ver ay % (- жа). 
Thus the unconditional distribution of U(t + 


€ п оп 1) is Gaussian, has mean 0 
and variance ао and the unconditional distri 


bution of U(t + т) is the 
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same as the equilibrium distribution of U(t). The joint distribution 
of U(t) and U(t + т) has the density 


2 


1 1 1 
= а ы EN OO 
fs = aces 9 (7 m2) vases 001—000] 
1 l i c ] 
“tava 8 P | Бера О 20А i}, 
where c=e-*', 

It follows that U(t) and U(t +7) have a bivariate Gaussian distribu- 

tion with 
E{U(t)} = E{U(t + I} = 0, 
соу {U(t), Ut+ =} = о0 e-?!*! - 
and var {U(t)} = oo. 

The mean and the variance of U(t) are finite and the covariance function 
cov {U(t), U(t +} isa function of the absolute difference only. Hence 
U(t) is covariance stationary. Again, since U(t) is Gaussian, U(t) is 
strictly stationary. | | 

Note that Wiener process is not covariance stationary. 

Example 6(b). The O-U.P. as a transformation of a Wiener process: 
Let X(t), t > 0, be a standard Wiener process. .Let 

1 
t)2 ——. X(a g(t), a> 0, 
"= Jt) CE 


and let the (non-random) function g(t) be positive, strictly increasing with 


g(0) = 1. 
We have E{Y(t)} = 0 
and var (Y) = ду; Vat a 800) 
1 E 
FO {a g(t)} = а. 
Since (X(t,),-- +» Хип) is multivariate normal, so also is (X(a Elt) 
aG Ar ҳа gt), Mast +9) 
cov (MO, Yt +9) = cor [rat 9 


= var (X(a a(t) H{(e() gu + DYF, 


since, for the Wiener process X(t), cov {Х@), X(t + х)} = var X(t) 


(see Example 5(b)). 
Now, var {Ха g(2)) = var (V g(t) YO} 
= g(t) var (Y(0) = а 800) 


and thus cov {Y¥(t), Yt a} = а (e(t)fe(t + D 


and for ¢ = 0, the covariance equals аа (х). 
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The process Y(t), which has finite mean and variance, will be covariance 
stationary provided cov (Y(t), Y(t ; =)} depends only on т. Thus we 
must have g(t -+ т) = 8(х).8(1), and in order to satisfy this equation, 
g(t) must be an exponential function, say, g(t) = e®™ (B > 0). We have 
thea 
cov (Y(D, Y(t + т) = а е0", (т > 0), 

and thus we find that for a> 0,8 >0 

Y(t) — e728" X(a e£) 
is a stationary Gaussian Markov process. In other words, Y(t) has the 
structure of an Ornstein-Uhlenbeck process. 


EXERCISES 


ES If X(t), with X(0) and y =0, isa Wiener process, show that 
Y(t) = oX (tc?) is also a Wiener process. Find its covariance 
function. 


5.2 If X(t) with ¥(0)- 0 and ы = 0 isa Wiener process and 
0 < s < t, show that for at least one т satisfying s < + < 1, 
Pr (X (т) = 0} == (2/2) cos! ((s/t)"*), 


5.3 Let X(t), with X (0) = 0, be a standard Wiener process and let 
T, be the first passage time of X (t). Show that T, and a? T, 


are identically distributed. If Zi, i — 1,2, ..., n are iid. as Т, 


then show that È Z;/ and Z, are identically distributed. 
5.4 Additive process: Here 
a (t, x)= a (t) and b (t, x) = b (t). 


Show that the Kolmogorov forward equation can be reduced to 


by a suitable change of variables, and that the corresponding 
distribution is Gaussian. 


Let X (t) be the displacement process corresponding to the velo- 
city process U (t) (O-U. process). Show that in equilibrium, 


E(X (t) — X (0) = 
and var (X (t) — X (0) =o? (gr — 1 + e-?Jps. 
Deduce that, 
var (X (t) -- X (0)) ~ o?1?/28 for small t 


7 c*t/8* for large t. 
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5.6 


5.7 


5.8 


5.9 


Show that if Y(t) isa O-U.P. with mean value 0 and covariance 
function C (Y (s), Y (t)} = ae-? !'^*!, then 
X (t) = (a]t)?? Y Q8 log (t/a)) 

is a standard Wiener process. 
Suppose U (f) is an O-U. process with E (U(t)) — 0, 
cov {U (s), U (D) = oe 7 and X(t), (where (d/dt) X(t) = U(t)) 
gives the position of the particle. 

Show that the conditional distribution of X (t) (t > 0) 
given that X (0) = xis normal with mean E {X(t)} = Xo 
and var {X(t} = (2007/8%) (ef — 1 + Вг) (Prabhu (1965)). 
Inverse Gaussian Process: Consider a Wiener process X (t) with 
positive drift. Without loss of generality, suppose that X(0)=0,, 
p=l,o=l. Then Te, the first passage time to a given point a 
(a > 0) has density 


exp {— (х — a)?/2x} dx, x > 0. 


fla, x) dx = —у бюз) 


Instead of a given point а, consider a variable point t (> 0). 


Then Т, has density 


This inverse Gaussian distribution has mean г and variance /. A 


new stochastic process having / for its distribution can now be 


defined. 
An inverse Gaussian process (or a first passage time distribution 


process) is a stochastic process {X (t) t > 0}, with X (0) = 0 
such that 

(1) X (t) has independent increments, i.e. for every pair of 
disjoint intervals of time (s, t) and (и, у), where s < t < u Ç v, the 
random variable X (1) — X (5) and X (v) — X (и) are independent. 

(2 Every increment X (t + /) — X(h) has inverse Gaussian 
distribution with mean / and variance f. 

For set, Е{Х()— Х(5)) = (1 — 5), 

var (X (t) — X (5) = (t — 5) 


and C (s, t) = cov (X (5), X (0) = var X (s)) 


(Wasan, 1969). 
Gamma Process: 
processes with zero mean value funct 


cov (Y; (5), Yi (0) = C1) for all i, 


If Yi (0, i= 1, 2, ..., m аге ii.d. Gaussian 
ion and covariance function 


then Y (1) = 5 Y? (1) is a Gamma process with parameter m/2. 
i=l 


Show that 
E(X (0) = àm Cl t), 


11(45-95/1982) 
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cov (X (t), X (s)} = фт (C(t, у)? 

and that the distributions of a Gamma process are gamma distri- 
butions. А р 

Negative binomial process: In our discussion on Poisson process 
we consider à as a constant or as a non-random function A(t) of 
time. It is natural to introduce some kind of random variations 
of A also; we may consider à to bea random variable or as a 
random function (or stochastic process) A(t). A Poisson process 
whose intensity function A(t) is a stochastic process is called a 
conditional Poisson process or a doubly stochastic Poisson process. 
Several studies have been made on conditional and mixed Poisson 
process (see also Sec. 4.3E). 

If the parameter A of a Poisson distribution is a random 
variable having gamma distribution, then the mixed Poisson 
distribution is negative binomial. By analogy, a negative binomial 
process N (t) isdefined as a mixed Poisson process whose intensity 
function à (f) is a Gamma process. It may be remarked that the 
distributions of a negative binomial process are not necessarily 
negative bionmial. à 

Suppose Y, (t) are iid. O-U. processes with E (Yj) = 0 and 
covariance function c (s, t) = o*e-81'-:1. Then $ i Y; (t) is a 

=] 


Gamma process and the mixed Poisson process, whose intensity 
function ^ (t) is a Gamma process, is a negative binomial process 
м). 

Examine- whether the distributions of N( 
bionomial. 

Show that 


f) are negative 
E {N (t)) = ymo 


and var {N (0) = фто? + mot (e-t — 1 + 29r)4g: 
(Barndorff-Nielsen and Yeo, 1969). 
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CHAPTER 6 


RENEWAL PROCESSES AND THEORY 


6.1 Renewal Processes 


Renewal process as a generalisation of Poisson process was introduced 
in Sec. 4.3 F. Here we propose to discuss the topic in some details. 

Let(X, n= 1, 2,...} be a sequence of non-negative independent 
random variables. Assume that Pr (Y, = 0) < 1, and that the random 
variables are identically distributed with a distribution function F-). 
Since X, is non-negative, it follows that E{X,} exists and let us denote 


EX) = J x dF(x) = y, 
° 
where р. may be infinite. Whenever џ = co, l/u shall be interpreted as 0. 
Lt  $,—0,$5,— X,-- X... Xo npl 
and let F,(x) = Pr (S, < x) be the distribution function of Sp. 
We confine here to the more important case of continuous X. 


Definition. Define the random variable 


N(t) = sup (n:S, « t}. (1.1) 

The process (N(t), t > 0) is called a renewal process with distribution 
F (or generated or induced by F). 

It is also customary to say that the sequence of random variables 
{Sm n = 1, 2,...} (as also the sequence TAL 368 
renewal process with distribution F. 

If, for some n, 5, = t, then a renewal is said to occur at t; 
the time (epoch) of the nth renewal, and is called nth re 
ration) epoch. The random variable N(t) gives the number of renewals 
occurring in [0, /]. The random variable Y, Bives the interarrival time 
(or waiting time) between (n — 1)th and nth renewals. The interarrival 
times are independently and identically distributed. When they have 


common exponential distribution, we get Poisson Process as a particular 
case. 


-}) constitutes a 


S, gives 
newal (or regene- 


Simple Examples 


One ofthe simplest examples of a renewal process is provided by lifetime 
distributions of a component such as an electric bulb which either works 
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or fails completely. Suppose that the detection of the failure of a bulb 
and its replacement by a new bulb take place instantaneously and suppose 
that the lifetimes of bulbs are i.i.d. random variables with distribution F. 
We then have a renewal process with distribution F. 

Consider a stage in an industrial process relating to production of a 
certain component in batches. Immediately on completion of production 
of a batch, that of another patch is undertaken. Suppose that the times 
taken to produce successive batches are i.i.d. random variables with distri- 
bution F. We get a renewal process with distribution F, where failure 
implies completion of production of a batch. 

The function M(t) = E{N(t)} is called the renewal function of the pro- 
cess with distribution F. It is clear that 
мМ) >п%® S! (1.2) 
Ма) > n ìf and only if Set 


te. 
Theorem 6.1. The distribution of N(t) is given by 
pat) = Pr {N(t) = n = F(t) — Кыа) (1.3) 
ег of renewals by 


and the expected numb 
Ми)= 2 К). (1.4) 


Proof: We have 
Pr(N() == Pr (N(t) > n) — Pr OP LEM 
= Pr {Sn < 1} — Pr «a < 0) 
= F(t) — Fuit): 


Again, 
M(t) = ЕМ) = B Pat) 


ES i (Ft) — Fr) 


2X Fit) 
nel 

= $ Pr{S St) Hence proved. A 
n=l 


4) can be put in terms of Laplace transforms as 


The relation (1. 


follows: ‘ 
Let F'(x) =Дх) be the density func 
the Laplace transform of a function g 
of both sides of (1 .4), we get 
13 so 
n-l 


M*(s)= B F- = 


tior (p.d.£.) of X, and g*(s) denote 
(t). Then taking Laplace transform 


2 NE VA 
2,2 VU" OT = sr £6) wi 
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This is equivalent to 
s M*(s) 
PO = rr. (156) 
These show that M(t) and F(x) can be determined uniquely one from the 
other. 


Note: M(t) = E(N(t)) is a sure function and not a random function 
or stochastic process. 


Example 1(a). Let Y, have gamma distribution having density 


ak xk-1 g-ax 


f(x) = E me x20. 


а NE 
f'())- (=) 
and the density F,'(x) of 


Then, 


$,— Х,+ X. Xn 


has the L.T. 
a nk 
(=) 
Thus Aen 
EE e arius] 
В) — ak — DI 
and hence 
x 
р) = | Fe’) d» 
0 
k—1 r 
=1— c e nl. 
Thus Polt) = F(t) — Feat) 


ы py (ex) 
r=nk r! 
and using (1.5), we get 
k 


M*(s) “Were: (1.7) 


Markovian case: In particular, when k = 1, X, has negative exponential 


distribution and the renewal process then reduces to a Poisson process. 
As is to be expected, we have then, 


pít) = е © 


and M*(s) = ajs? sothat M(t) = at. 
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Example 1(b). Mixture of exponential distributions. Let X, have density 
A(t) = рае“ + (1 — р) be*', 0< p < 1, 

a>b>0. (1.8) 

Such a model may be used to describe a system which has two kinds 

of components—a proportion ‘p’ of components having a ‘high failure 


rate a and the remaining proportion (1 — p) of co t à 
lower failure rate b. We have шропеп Шаша 
eee fel Ыр) b 

fO YER (1.9) 
so that 
ab + s [pa + (1 — p) b] 
2 [5+ (1 — p) a - pb] - 
Writing A — pa 4- (1 — p) b and B= (1 — p) a + pb, we get 


As + ab 
М") = 3 GEB) 


M*(s)= 


ee T ао2 а 
=s(s+ В) s(-B) 


Б }+2 4. 1 
= {з staf B (9 {5-32 


Inverting the L.T., we get 
B uL. 2í-—E 
M()- 4 0-€9)* 1-7 в } 


abt 
=F + C(1— e, (1.10) 
where 
A. ab AB— ab 
gw ($ pB] ^ m5 


-20 =) а?) >0. 


When р = 1 (or р = 0) the distribution of X, reduces 
ial and then C — 0, i.e. the second term of (1.10) 
) = at (or bt). 


Markovian case: 
to negative exponent 
vanishes, so that we get M(t 


6.2 Renewal Equation 
An integral equation can be obtained for the renewal function 
M(t) = E {ND} > 


which gives the expected number of renewals in [0, 1]. 


nction M satisfies the equation 


Theorem 6.2. The renewal fu 
1 


ми) = Ки) + | M (t — x) dF (x). (2.1) 


в 
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Proof: By conditioning on the duration of the first renewal X,, we get 
M(t) == E (NQ) = | E {N(t) | Xi = x} dF (x). 
0 


Consider x > t; given that X, =x > t, no renewal occurs in [0, ¢], so 
that 
E {N(t) | X1 = х) — 0. 


Consider 0 < x < t; given that the first renewal occurs at x (« 1), then 
the process starts again at epoch x, and the expected number of 


renewals in the remaining interval of length (t - x) is E (N(t — x)), so 
that 


E {N(t) |X, > x] 1 E(N((— 3) 
=] + M(t—x). 
Thus considering the above two equations, we get 


t 


ми)= | a + Ми — 3) dF (x) 
0 


=. F(t) +- f M(t — x) dF (x). 
0 


We have thus established (2.1). A 


The equation (2.1) is called the integral equation of renewal theory (or 
simply renewal equation) and the argument used to derive it is known as 
‘renewal argument’. The renewal equation is also expressed as 


M = F+ Ms. 
The equation (2.1) can also be established as given below. 


Webs Ми)= È Fan RO È Fa) 
a= n=) 


= F(t) + H ij Е, (t — x) dF w} 


F,4, being the convolution of F, and F, = F. Thus 
t 
мо) = FO) + |{ È 5,0 0) areis 


9 


assuming the validity of the chánge of order of integration and summa- 
tion, we get 


M() = FO) | Mu =x) ar 3. 


0 
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It follows that M(t) = 2 Р, (1) satisfies the integral equation (2.1) 
п=1 . Seo 


The renewal equation (2.1) can be generalised as follows 
t 
v(t)=g(t) JE (t — х) dF (x), t > 0, (2.2) 
0 
where g and F are known and v is unknown. The equation (2.2) is 
called a renewal type equation. 
A unique solution of v (t) exists in terms ofgand F as can be seen 
from the following: 


Theorem 6.3. If 


(Dg) ID (t = x) dF (х), tz 0 (2.2) 
0 

v(t)— (0) rfeu-» dM (x), (2.3) 
0 


then 


where M(t) = i Е, (0). 


Proof: Taking L.T. of (2.2), we get 
у* (5) = g* (5) + v* G) /*(65) 
or, у (Sys "OS 
ou S) 
-rolit Fo] 
= g* (s) [1 -+ sM* (s)]. 


Inverting the L.T., we get А 
=a) + [20-9 4M co 
0 


and the solution v(/) is unique, since a function is uniquely determined 
by its L.T. Thus the theorem is proved. A 
For examples of phenomena satisfying a renewal type equation, see 


Feller II (Ch. XIV. 2). 


6.3 Renewal Theorems 


For the Poisson process which is a renewal process with exponential 


interarrival times X». 
M(t)= at 


or, M(t)/t = а = VEG). 
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In the general case, the result 
м0). Ip, B= ЕХ) & oo 
Feller) is known as elementary 
d t- œ. The result (due to eller) r : 
ке theorem. Before considering this, we consider a simple result. 
Theorem 6.4. With probability 1, 
AD 1 as t— oo, (3.1) 
г р 
where u = E(Xn) < ©. 
Proof: Consider an interval [0, t], we have 
Sunin &t < Saa (3.2) 


Now the strong law of large numbers holds for the sequence {S,}. so 
that, as n — со 
Se ep Cee es 
n п 


with probability 1. Again 


> Е(Х,) =u 


as t+, N(t) co 
with probability 1. Thus, with probability 1, 


MO?” as г co. (3.3) 
Similarly, with probability 1, 


Sven | Snwar | M(t) 1 


NO) N41 —Ngy 5851 o. ы) 
Thus from the three relations, we get that, with probability 1, 


N(t)/t> l/past>o. А 


This theorem shows that for large t, the average number of renewals 
per unit time converges to l/u. 
We now go to the renewal theorems. The elementary renewal theo- 


rem is due to Feller (1941); here we shall follow the arguments put 
forward by Smith (1958). 


Wald's Equation: Before going to renewal theorems, we discuss a 
special type of non-negative r.v. associated with a sequence of r.v. {Xj} 
(or a stochastic process {X(t)}). Such a variable, first considered by 
Wald (1947) while formulating sequential analysis, is known as а r.y. 
inde pendent of the future or a Markov time or a stopping time. For an 
elaborate discussion on Markov time and its applications, see A.N 
Sirjaev (1973). We consider a somewhat special case here. «i 
Definition. An integer-valued random varible N is said to be a 


stopping time for the sequence (X;) if the event (N — n) is independent of 
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Жан» Anto Хата: -- for all gb == 1,2; .... 


It can be shown that N is a proper random variable. 

As an example of such a variable, consider a coin tossing experiment 
and the result X; of the ith toss with X; ==-1 or 0 depending on the result 
being head or tail respectively. 

The sum 

Sa Xp rb Xn 


denotes the number of heads in the first л toss. Suppose that m isa 
given integer, then N = min {л such that S, = т} is a stopping time. 
Consider the number A(t) of renewals by time /, w.r.t. a sequence of 
interarrival times {Xj}. Now M(t) = n whenever Sn < t and Sy, > t, i.e. 
the: event N(tr) = п depends not only on X;, ... , X, but also on Xpy 
Consider the variable N(f) -- 1. Now N(t)+ 1 = n implies S, , < t and 
S, > t, so that the event (N(1) +.1 = n} is independent of Y,,,, 43, .... 
Thus N(t) + 1 isa stopping time for the sequence {Xj} while N(¢) is not. 
From a corollary to Theorem 1.3 we find that: if 


N 
Sy E X, 
\ i=1 
where X, аге i.i.d. random variables and N is a r.v. (independent of 
Хз) having finite expectation, then 


E (x Xj = EX) E(N). 


The same result holds also when N is a stopping time for the sequence 
TED The proposition which is used in proving certain results in renewal 


theory is given below. | 
О eovorion : If {Xj} is a sequence of i.i.d. random variables and 
N is a stopping time for (Xj) and E(N) « со, then 
- Е(Ў X} = EQ E {N} 
i=l 


1 I Wel 
а-| 


Proof: Let Om af NI, 
N = : 
so that 200 zi men 
N vy реў = SEE NN 
and thus E ta X}=E (225 2) Es млд 


dity of the change of order of expectation and sum- 


assuming the vali Ur Жу. Nn. amd de 


mation. Now Z; is determined by {N < i}, 
thus independent of X; Thus 


E (5 х} = EE {Xi} EU 
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= E {X} X EQ) 
i=l 


= E(X) È PUNTO 


= E {X} E {N}- 
Hence proved. A 


Corollary: Since N(t) + 1 is a stopping time for the sequence xi) 


we have, 
кау+1 
Е (Sinai =Е{ LA Xy 


is’ 
= E {X} E{N(t) + 1} 
= E {X} (M(t) + 1}. 


Theorem 6.5. The Elementary Renewal Theorem. We have 

M(t) a 1 

t u 

where u = E(X,) < co, the limit being interpicted as 0 when p = со. 
Proof: Let 


as to, (3.5) 


N=N(t)+1 and Sy=t i И), 
where Y(t) is the residual lifetime of the unit in use attime t. Let (X;'), 
} = 1; 2,...}, bea sequence of independent realisations of the renewal 
process {Xj}, 5,0 be the corresponding partial sums, and NUXt),the 
corresponding number of renewals in [0, t]. 
Let «= Sy, My = NOt) 4-... c NOC), 
Ty = tO) 6...5 800, 
Now Гу is the sum of (k - Mx) i.i.d. random variables-X;. Thus, by the 
strong law of large numbers, if ы = Е(ХІ) < со, then as К-> co 
Mk | 
E edu. 
к= М К+ M, ^ (3.6) 
with probability 1. 
By the same law, we get, as k — oo, 


k 
, END 
ј=1 
р END = ми) (3.7) 
Ў xo 
апа Tk = Ja) 3 


«X c EO = Ett + Y(t} 
=t+ ECY(t)} (3.8) 
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Combining (3.6), (3.7) and (3.8), we get, as k > со 
Tk zE P me) st + eae) ША 


with probability 1. 


kM, k IFMA 
or, t+E Y(t) =p{l + MO} 
Че. 4 FOO) {3 ©, 
and now Y(t) is positive, E(Y(t)) finite and hence as t — оо 
E Deel 
lim inf —— =-. 
at ЕГ 2:9 
To prove ће theorem we have to show that 
ME 
li 
en [rinm 


For doing this we define a new renewal process as follows: 
Let A be a constant > 0 and for n = 1, 2, ..., and let 


x= xe MF и \ 
" in AR if) Xs 27А 
Let е 2 X,* and N*(r) = sup(n: S,* < t), 


M*(t) — HUN 
S: «S, and hence .V*(r) > Nt) 


Now 
and М*() > M(t). 
Араїп Е(Х,*) = ua <и and pa p as А > о. 
We have 
Shon © t £ 
E (Shas) = UA (M*(0 + D 
and hence 


M*(t) 2 1 
== <— 
Шш PEE "л 


and as A> со, 
Mi 
lim sup —— мш det i (3.10) 
>= 


From (3.9) and (3.10) the result follows. A 


Example 3(a). Age and block replacement policies: 
(Barlow and Proschan, 1964) 


The usual replacement policy implies replacement of acomponent, as and 
when it fails, by a similar new one. There are other policies besides this; 
the two most important replacement policies that are in use, in general, 
are the age and block replacement policies. Under an age replacement 
Policy, a component is replaced upon failure or when it attains a specified 
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age T, whichever occurs earlier. Under a block replacement policy, a 
component is replaced upon failure and also regularly at times T, 2T,.... 
We shall call 7, the replacement interval under these two policies. We 
assume, as usual, that components or units fail permanently, indepen- 
dently and that the detection of a failure and replacement of the failed 
item are instantaneous. Suppose that the successive lifetimes of the 
units are random variables with a common distribution function F(x) 
having mean y. Considering failures as renewals, the number of failures 
in [0, t] under the three replacement policies, the usual one, the age 
replacement and the block replacement can be denoted by three renewal 
processes as indicated below: 
N(t) — the number of failures in [0,7] for an ordinary renewal 
process. 
Na‘t, T) = the number of failures in [0, /] under age replacement 
policy, with replacement interval T. 
Na(t, T) = the number of failures in [0, 7] under block replacement 
policy, with replacement interval T. 
Let the corresponding renewal functions be 
M(t) = E {N(t)} 
Malt, T) = E (Na(t, T)} (3.11) 
Мв(ї, Т) = E {Na(t, T)}. 
It will be shown here (see Barlow and Proschan) that 


lim. Natt, T) = lim = = Sho) MN (3.12) 
fa — F(x)} dx 
0 

; Ny(t, T) t M(t, T M(T 

CINEMA er 6.9 


To prove (3.12) we first note that (NA(t, Т), > O} is a renewal process 
with distribution 


Fat) = Pr (Y; & t) 
such that, in 
nrecicrt)s5-0.1.2..., 
1 — Fat) = Pr (Y; > t) = [1 — KT) [i — F(t—nT). — (3.14) 
Hence ' 


va = EY) | Prod 
(п+1)г 


=3 | -ATP Fe- nT d 
2 T 


n 
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Ж 
-3 [RTJ f a— 5641 
$ 0 
а | (1— Коў} dx/F(T). (3.15) 


0 

With this value of wa, using Theorems 6.4 and 6.5, we at once get (3.12). 

To prove (3.13) we notethat after successive block replacements 
at T, 2T, ..., the process starts anew, and if N5,(T) is the number of 
failures in the interval [(r — 1)7; rT], r = 1, 2, 3,. .., following block 
replacements, then №в,(Т) = N(T). 1 

Suppose that for some specified integral value of л, nT < t < (n + 1)T, 
i.e. for some n and 0 <т < T, 


t —nT 4-3, 
then 
Ng (t, T) = у, Мв, (T) + №) 
I 
= È М(Т)+ NO). 
r=1 
We then get 
п 
У Nt) 
lim Мв (1, Т) — nm 
1 t пэ = = 
= E{N(T)}/T, (by the law of Large Numbers) 
= М(Т)/Т. 


Thus we get (3.13). A 


Particular Cases з 
(i) Markovian Case: When F(x) = 1 — e~*/* with mean p = a, then 


T 
wax | 0- FON 6/80) а= 
0 

and M(T)T = aT/T — a = e; 
as is to be expected (Explain why?). 

Gi) Limiting Case: Suppose that y. is finite, then for large T, we have 
asymptotically, m re 
and by Theorem 6.5, M(1)/T > ly. 

It should be intuitively clear that for T-> оо, the renewal processes 
under age and block replacement policies are both equivalent to the 


ordinary renewal process N(/). For some interesting properties and 
renewal theoretical implications of these replacement policies, refer to 


Barlow and Proschan (1964, 1975). 
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1 
We shall state without proof two other renewal theorems hereafter. 


Definition. A discrete random variable X is saidto be a lattice 
variable (or to have a lattice distribution) if it takes values c+ nd (n — 0, 
+1, +2, ...), where c and d (>0) are constants. The largest d is 
said to be the period of the distribution. (Feller usesthe terminolog 
‘arithmetic’ for lattice and ‘span’ for period but the terminology used 
here is more common.) 
Lattice variables form a special type of the class of discrete variables. 
When c= 0, d=1, a lattice variable becomes an integer-valued variable. 
The classification of random variables is given below: * 


Random variable 


Discrete Continuous 


m 


Lathice Non-lattice 


Integer vatued Non integer valued 


By saying that a distribution is nor lattice, we shall mean that the 
corresponding random variable is other than lattice, that is 


‚ it is either 
discrete non-lattice or continuous. 


Theorem 6.6. (Blackwell’s theorem): For X; not lattice and for fixed л 20 
M(t) — M(t — h) > hlu as г oo 
and for a lattice Y, with period d, 


lim Pr {renewal at nd} > аи. 
пэ ә 


A more general form of the theorem is given below. 


Theorem 6.7. (Smith's theorem or the Key Renewal theorem): 
Let A(t) be a non-negative, non-increasing function of t > 0 such that 


[о dt < oo 
0 


and X; be not lattice, Then 


1 t 


| h(t — x) dM(x) > | h(t) di, 


0 


as f — со 


о 
the limit being interpreted as 0 when и = %. 


Ж A r.v. may also be of the mixed type, being partly discrete and partly con- 
tinuoüs, 
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For proof of Blackwell's and Smith's theorems, see Prab 
Feller (1966). н: 
. Smith's theorem is called the key renewal theorem as a number of 
important results can be derived from it. For example, if 


Д) А, O0O-ct«h 
= 0, otherwise; 


We at once get Blackwell's theorem 
h 


M(t) — M(t — h) > 2 [а = hip. 
* 7$ 


6.4 Delayed and Equilibrium Renewal Processes 


The renewal process {А7} which we have considered so far is such that 
all the Хр, і = 1, 2, ... are iid r.v.'s The process is what is known 
as ordinary renewal process. There are two generalisations of the ordinary 


renewal process. 
First, suppose that the first interarrival time X, (i.e. time from the 


origin upto the first renewal) has a distribution G which is different from 
the common distribution F of the remaining interarrival times X,, X,, ..., 
i.e. the initial distribution G is different from subsequent common distri- 
bution F. We then get what is known as a modified or delayed renewal 
process. Such a situation arises when the component used at t = 0 is 
not new, When .G = Е, the modified renewal process reduces to the 


ordinary renewal process. 
Definition. Let (X; i = 1, 2, ...} be a sequence of independent non- 
negative random variables such that the first interarrival time X, has the 
distribution С and the subsequent interarrival times X,, n= 2, 3, ... 
have the identical distribution F. 
Let 

y YI y 2 X; and Np(t) = sup (n: S, < t). 

i=) 


The stochastic process (Np(!), t > 0} is called a modified, delayed or a 


general renewal process. , » 
The sequence of random variables {Xp п = 1, 2, ...}, where the initial 


distribution G is different from subsequent common distribution F, is 


said to constitute a modified renewal process. 
The following results can be easily obtained: 


We have 
Pr {Np(t) = n} = Pr {Sn < 1) — Pr {Snir < t) 


= G&F,a(t) — G*F,(t) 


50 that = 
M(t) = E {N} = Z, б*Р,-1(0) (4.1) 


12(45-95/1982) 
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and e “a 
Mp*(s) = st —f*Gy (4.2) 
g* and f* being the L.T. of the p.d.f. of G and F respectively. 
Again, М 
мыр) = f E (p(t) | X, = х) dG(x) 
2 | {1+ M(t х) dG() 
= a(r) + | ми х) 4009); (4.3) 
0 


M(t) being the renewal function of the ordinary renewal process with 
distribution F. 


Example 4(a). Consider a modified renewal process with the first inter- 
arrival time X, having mixed exponential distribution (considered in 
Example 1(b) and the other interarrival times X,, X, ... having simple 
exponential distribution with parameter b. Such a model would be 
suitable when there is no defective item among replacement parts (i.e. 
whereas X, has distribution as described therein, the distribution of 
Xa Xa.. corresponds to p = 0 in the distribution of X,. Such a situa- 
tion has been visualised by Karmarkar, 1978). We have 


g(t) = pae + (1 — p) bet, a > b> 0. 
From (4.2) we get 
ees) 
MONO = S STA 
_ab+{b+p(a—b)}s 


s* (5 +a) 


= ab 4 b p (a — b) 
s*(s+ a) 5 (s+ a) 


Inverting, we get 


M(t) EL І e жора а 


=и+”@—9® og- =e), 
M/(t) can also be obtained directly from (4.3), by noting that the 
renewal function M(t) of the ordinary renewal process with distribution 
F is given by 
M(t) = bt 
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and G(t) = [ g(x) dx. 


We state, without proof, the corresponding renewal theorem for 
modified renewal process. 
| 


Theorem 6.8. Let | р= Е {Х,}, п= 2, 3, .... Then 


(1) with probability 1, 
Mpo(t)/t > le as t—> œ. (4.4) 


(2) If Fis not lattice, then 
Mp(t) — Mp(t— a) > alp. as t — © ‚ (4.5) 
and if F is lattice with period d, then 
Pr (renewal at nd) > dl. as п > oo. 
The above theorem shows that the renewal rate tends to a constant 
and is independent of the initial distribution G. We have, as Feller 
Observes, an analogue to the Ergodic theorem for Markov chains. 


Now M(t) > tly raises aquestion: whether there exists an initial 
distribution G such that Mp(t) = t/u. The answer is in the affirmative. 


This is true in case of a kind of renewal process discussed below. 


Equilibrium (or stationary) Renewal Process 
Now a modified renewal process with initial distribution G — F, exists 


such that 
Mt = Уы, 

ог M.A(s) = Mus? 
F eid 
iff f(s) = LO. { 

pela F(x) 
or f= 
ie. F(t) aj ше dx. (4.6) 


0 
F(t) is called equilibrium distribution of F 
] process is called equilibrium (or statio- 
ilibrium process by {N.(t), t > 0} 
n by Mt). Poisson process is 


The distribution having d.f. 
and the corresponding renewa 
nary) renewal process. We denote an equ 
and the corresponding renewal functio 


clearly an equilibrium renewal process. 
Equilibrium renewal processes, which are extremely important in appli- 


cations, can ariseas follows: Suppose thatan ordinary renewal process, 
has been continuing from time f> — 9? remote from‘ = 0. If the 
Observation of the process is started at / = 0. then the interarrival time 
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X, to the first renewal will have distribution function G — F, as given 
by (4.6). 


Example 4(b. We have, for all t > 0, and n =O, 1, 2, .:. 


Pr (Nt) > n} > Pr {N(t) > n}, 
iff 


| (1— Ао) dx 
крсу es 


Proof: Assume that the condition holds. Since 


æ © t 
1 de Er P PES 
1 | {1 — Дх) dx = aljo F(3)) dx Ja FG) dx | 
=1—F,(t), 
we have by hypothesis, for £ > 0 
| — F(t) < 1— Ft) 
or, F(t) > F(t). 


If S, and 5, denote the nth renewal epochs of N(t) and N,(t) respec- 
tively, then, for n > 1, 


Pr {NAt) > п) = Pr (Sj < t) 


Е"-1 (t — x) dF.(x) 


2| FO (t — x) аҢх) 


2—5 on 


= Pr (S, < t) = Pr (N(t) 2 n). 
Further, Pr {N.(t) > 0} = Pr (N(t) 2 0} = 1, for all t 2 0. 
result holds. 
To prove the converse, assume that Pr {N,(t) > n) > Pr {N(t) 2 n} 


holds for n= 0, 1,2, .... Since equality holds for n —0, we must 
have 


Thus the 


Pr (№40) = 0) < Pr (M(t) = 0) 
or, куйшде] 
and therefore the converse is also true. Д 


Corollary. M(t) < t/y iff F(t) > F(t). 


Note: We describe two concepts used to classify distributions in 
reliability theory. 
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A unit (or equivalently its lifetime distribution F) is said to be New 
Better (Worse) than Used NBU (NWU) if 
{1 — Fx) < (2) К Fo) 0 FON. 
In other words, a unit is NBU (NBW) if the survival probability of a 
unit at age x is less (greater) than the corresponding survival probability 


of a new unit. 
A unit is said to be New Better (Worse) than Used in Expectation 


NBUE (NWUE) iff 


f {1 — F(x)} dx 


I—FD <(>) E 


In other words, a unit is NBUE (NWUE) if the conditional expectation 
of the residual life of a unit at age / is less (greater) than the expectation 


of the lifetime of a new unit. 
The result given in the above example may thus be restated as: 


Pr {N.(t) > n} > Pr{N(t) > m, t 20, n=0, 275.5 
as well as M(t) — tlw, iff the distribution F inducing the renewal process 
is NBUE. А 
Probability Generating Function (p.g.f.) of Renewal Processes: 

We obtain here the p.g-f. of the renewal processes 
{N(t), t > 0} {Np(t), t > 0), (000), t > 0). 


Let к, 
са, 2) = E 7 РЕМ) = r} 


= b zr (F, (t) — Fra (0) 


r=0 
=14 $ z- (z— 1) F(t). (4.7) 
r=1 è - 
The above relation holds for all the three types of renewal processes. 


We denote the p.g.f. of ordinary, delayed (modified) and equilibrium 
renewal processes by G, Gm and С. respectively. Now the Laplace 


transform of the p.d.f. of F, (for r > 1) is 


fe) = LAG. - (for an ordinary renewal process) (4.82) 
—g*(LA* GI (for a delayed renewal process) (4.8b) 
(4.8c) 


екш ые 
T (for an equilibrium renewal process). 


Denote the L.T. of the p.g.f. of the ordinary, modified and equilibrium 


Tenewal processes by 
G*(s, 2), Gn*(Ss 2), Ge*(S: 2) 
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respectively. From (4.7), we have 


оҷ =1 +22 202 — 0U*GY 


= = oe (4.9) 
2 me- 1) 9 U*Gy? 
(z— 1) g*(s) 
s{l—zf*(s)} 
ўеш йш, em 
15 eag- 78) pe 


1 
5 
1, 6— Dü —f*G) 
$ 
1 


M 


d 
S 
Д 
5 
1 


us {1 zO) (4:11) 


=> ES —- G*(s, 2). (4.11a) 


Inverting (4.11a), we get 


t 
Gt, z)=1+ em ви, z) du; 
0 


comparing the coefficients of z” from both sides, we get 


t 


Pr (N41) = г} = 1 || [Рг (N) = r— 1) — Pr (NQ) = r]du (4.12) 


j 


| Pr (N(u) = 0) du. (4.122) 


1 
ртм) = 0) = 1-7 


Example 4(с). Second moment and variance of N(t). 
The moments of A(t) can be found from the L.T. of the p.g.f. Howe- 
ver, the following simple procedure may be employed to find the second 


moment of N(t) and thence its variance. We have 


Lt) = ENOJ = È (Ft) - Faso). 
Taking L.T., we get 


10) = X SU*9y (Учун) 
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On simplification, we get 

FOUE 
s{l—f*(s)¥ 
LERO] ey 


L¥(s)= 


Б [ —f*(s) 
which can also be written as: 


L*() = M*()4- A ОО 


Inverting the L.T., we get 
Ем = 10 = м0 + 2 | ма x) амо) (4.13) 


0 


This gives the second moment of N(t) in terms of the renewal function 
M(t). Thus var {(N(t)} can be obtained in terms of M(t). 


Example 4(d). An application to а management problem. For certain 
kinds of electronic equipments of office use (such as electronic calculators 
etc.) and of domestic appliances (such as T.V., refrigerators etc.) the 
manufacturer offers to the customer a certain kind of service contract by 
which the manufacturer undertakes to maintain and repair the product 
ed period against payment of a fee by the customer. 
оп regarding the offer of the terms of contract 
fees to be charged thereof), the manufacturer 
pected cost of future repair. A simple renewal 
Karmarkar (1978) is considered here. It 
followed by instantaneous detection and 


over a certain specifi 
In arriving at a decisi 
(including the amount of 
should be aware of the ех 
theoretic model developed by 
is assumed that each failure 1s 
Tepair. 
Denote 
R(t) — cost associated with a 
that R(t) = Ё, 
c =the discount factor for 
costs, І 
f(t) = the p.d.f. of tim 
‘k,(t) = the p.d.f. of S, = Ал 
V, = the discounted cost of rth renewal. 
gle failure followed by renewal, the expected discounted cost 


repair (renewal) at time ¢; it is assumed 
“continuous” discounting of future 


e to failure (interarrival time X;), 
+X; F m df. of S, 


For a sin 
E(V,) is given by 


E(V) = [ Re~ f(t) dt = ЕЁ /*(с). 
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The expected discounted cost for the rth renewal is 
EW.) = J Re“ k(t) dt 
0° 


= Rk,*(c) 
= RION. 


For a sequence of renewals, the total expected discounted cost is 


E(V) & EV, + Vat V) È RIPON 


RI ("or 


Ife) 
К f *(c) 
ла asn-- su (4.14) 


For a sequence of infinite renewals truncated at time 2 the expected cost 
of future renewals is 


Р ia 
E(WT)-— Z | Re k,(t) dt 
r=] o 
T 
= f Re  $ F(t) 
Н т=1 
т 
«| Re~ dM(t), (4.15) 
0 
assuming the interchangeability of summation, integration and differen- 
tiation. As T — œ, 
RF*(c) 
E(V(T. RcM* (c) = M 
UT)» ReM* (с) =- 7%) (4.16) 
which agrees with (4.14) asn > оо. 


Particular cases ° 
(1) Suppose that f(t) = be, then 


b 
ip o 
-(S)h-( b | 
с all 
R 
= as п» со 
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and for infinite renewals truncated at Т, we get 
T 


ET) = [Ree d(bt) 


0 


| 
as TA o. 


(2) Suppose that f(t) = pae™™ + (1 = р) Бе, 0 <р< 1, а> Ь> 0, 
ie. Y, has a mixed exponential distribution as discussed in Example 
l(b. Then 
ч. pa ,(1—p)b _ {ра ,(0—p)5 
ву) R[F + c+6 1- {2а e+b 

. R(Ac + ab) 
ENLCLIJE 
A—pa--(1—p b, B—( — p) a+ pb, 


where 
3 | 
ab, [А ab 
вит) | {+ ($-a) 2} 
0 


Rab R(AB— ab) r „дев 
= (7t Bc B) Ше е], 


(3) Suppose that 
Att е“ f > 0 


Kt) = T) 
so that é 
XE 
f*(s) = (23) , 
then . 


R* 
EV) 17 JE х АЕ 


i i So far we have 

: Number of renewals inva random time. х 
тр number V(t) of renewals in a fixed interval [0, t]. Consider 
now that Т is arandom variable independent of X; so that 10, T]isa 
random interval; suppose that N is the number of renewals in [0, 7]. 


Let G(s, 2) be the p.£f. of N(t) and G(z) the p.g.f. of N. If q(t) is the 
p.d.f. of T, then it is clear that 


G(2) = fee z) q(t) dt. (4.17) 
0 
is Sorek that T has exponential distribution, with p.df. 


q(t) = a&-^. Then 
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G(z) = f ае“ G(t, 2) dt 
0 
a 


G* (a, z). 
If the renewal process is ordinary, then from (4.9) we get 
G(z) — aG* (a, z) | 

a (1 — f* (ay 
~ atl —z* (2j 
A ecd CA 
lito 

which is the p.g.f. of a geometric distribution. 

We have 


Pr (N =r} = [/* (ау [1 —f* a], r=0, 1,2, .... | 


Suppose further that the renewal process is Poisson with S(t) = be; | 
then 


a 
Ce) Tr 


and Pr = (2) (2). 0, (#2 oss 


This result was obtained in Ch. 4 (see Poisson count process). 


(2) Suppose that the renewal process is Poisson but T has gamma 
distribution having p.d.f. 


a(t) = ak t? ec"lT (k), 12:0; 
Т can be considered as the sum of k i.i.d. exponential distributions with 
p.d.f. q(t) = ae", t2 0. 


From the memoryless property of exponential distribution, we then get 
that the distribution of N is negative binomial (Pascal) with p.g.f. 


(ae - | 


a k 
=a 


_ a (*t*r-1 а ҮЕ b ү 
а Ges o ra orm a. 


6.5 Residual and Excess Lifetimes 


so that 


We discuss here two random variabl 


d es of interest in renewal theory. 
To a given t > 0, there corresponds un 


iquely a N(r) such that 
Snin € t < Syoya, 
ie. t falls in the interval Xy. 
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(i) The residual lifetime of the individual alive at age t is given by 
the time Y(t) from t to the next renewal epoch, i.e. 
Y(t) = Swen he 
It is also called forward recurrence time at 1. 
(i) The spent lifetime of the individual alive at age г is given by the 
time to f since the last renewal epoch, i.e. 
Z(t) = t — Su. 
It is also called backward recurrence time at 1. 
(iii) The length of the lifetime containing ¢ is given by 
Y(t) + Z(t) = Sma 7. Swe = Xnr 
The distributions of Y(t) and Z(t) are given below. 


Theorem 6.9. We have 
Pr (Y(f) < x} = Ft +x) — f [1 — F(t + x— у)] 4М(у). (5.1) 


0 


If in addition, F is not lattice, then 


x 


[п — FJ ау. (5.2) 


1 
lim Pr {¥(t) < x} ei 
ipa о 
Proof: Itis clear that 


Y(t) > x « no renewals in [t, t + х]. 
Let P(t) =Pr{¥(t) > х). 


By conditioning on Ху, we get 
P(t) = Pr {¥(t) > х} 


= [prato > x13 75 dF). 
0 
‚ Now three situations may arise: @y>tt+ x, in which case it is certain 
that no renewal occurs in [t, £ + x], so that 
Pr (Y(t) > x} = Pr (no renewal in [t, t + x]) 
=1 


(i) ray <t+x,in which case one renewal (the one corresponding 
< 


to X, = y) occurs in [/, t + x], so that 
Pr {¥(t) > х) = Pr {no renewal in [t, t + xpo 
(iii) 0 < y < t, in which case the first renewal occurs in y, y 4- dy and 
then the process restarts itself and 
Pr {¥(t) > = Pr {¥(t— у) > x} 
=P(t—Y). 
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Considering the above three situations, we get 


t o 


р = | раа) + | aro» 
° thx 
t 
or, PQ) =1—F(¢+x)+ | Р@— у) dF (у), 
Е 0 
which is a renewal type equation. Applying Theorem 6.3., we have 


P(t) 21— F(t4- x) + [a-recx-») dM(y), (5.3) 
0 


which is equivalent to (5.1). Thus (5.1) is proved. 
Now suppose that F is not lattice. Then applying the key renewal 
theorem (Theorem 6.7) with h(t) = 1 — F (t + x) we get, as t-> оо 


Ju- (+ х— amo) = Ju- ra) a 


1 
u 


=if [L— F 9)] dy. 


Hence from (5.3), we have 


lim Pr (0 > 3) = lim РО) == fü — Foy dy 
LJ 


or, lim Pr {¥(t) < xj =1 -i fu- ro» dy 


x 


=] — коў. 


LU 
Thus (5.2) is proved. A 
Theorem 6.10. We have 
1-х 
ғ0)— [a —F(t— y) ам <t; (5.4 
нао А ОЕЕО 
1 X'S. (5,5) 
If, in addition, F is not lattice, then 


x 


lim Pr {Z(t) < x} =7| {1 — Е(у)} dy. 
0 
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Proof: The distribution of Z(t) сап be obtained immediately by 
noting that 

Z(t) > x < no renewals in [t — x, t] 
= Y(t— х)-> x, 

So that Pr (Z(t) > х} = Pr {Y(t — x) > x), 

Hence from the Theorem 6.9, we get 

Pr (Z(t) < x) = Pr {Y(t — х) < х) 


1-х 


=F) – | 1-5-0) 4м0), x <1 
0 
=] x>t 


and if F is not lattice, then 


lim Pr {Z(t) <x} = | (1 — FOX dylu. А 


omx 


Note: It is to be noted that the result 
x 
lim Pr (Y() < x} =f {1 — FO)} dyle 
too 0 


г holds also for the modified renewal process regardless of the initial 
distribution G. 

Now the R.H.S. is the distribution function F, = G of the equilibrium 
renewal process which has a uniform renewal rate M(t)/t = y. Thus if 
V. < co, then the residual lifetime Y(t) has a distribution which coincides 
with the distribution of the process attaining uniform renewal rate. 

If u= оо, then the probability that the residual lifetime exceeds x 
tends to unity for all x. This should be intuitively clear. : 

The asymptotic behaviour of the residual lifetime distribution has been 
investigated at length by Balkema ef al. (1974) who obtained the possible 


limit distribution types. 
Example 5(a). Markovian case: Let X; have exponential distribution 
with distribution function F(x) = 1 — е7“. 


Then M(t) = at, and we have 
Li 
procy- denm [ (1 — 1+ emm da» 
0 


—]1-—e-^ for all t. 
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Thus Y(t) has the same exponential distribution. In fact this is one of 
the characteristics of exponential distribution. 


For x <4 
fox 


Pr {Z(t)< x} = 1 — е7“ — | {e7} d (ay) 
0 


== eo 
and for x >f, 

Pr (Z(t) < x) = 1. 
For large t, Z(t) will also have the same exponential distribution. 

From the memoryless property of exponential distribution, it is clear 
that Y and Z are independent. As 
Xvi = Y(t) + Z(0, 

the renewal interval containing ¢ is given by the convolution of the 


residual and spent lifetimes at t. For large f, the distribution of Xna 
will be gamma with mean 2/a, i.e. 


E (Xni) = 2/a = 2Е(Х)). 


Thus for large t, the expected duration of the renewal interval containing 
t will be twice the expected duration of an ordinary renewal interval. 


6.6 Renewal Reward Process 


Consider an ordinary renewal process generated by the sequence 
{Xn n—1,2,...) 
of interarrival times. Suppose that each time a renewal occurs, a reward 
is received (e.g. for the occurrence of the renewal) or a cost is incurred 


(e.g. for replacement of the failed component). Denote the reward or 
cost associated with nth renewal by 


Ya nz 15235... 


Now У, will usually depend on X,, the duration of the interarrival time 


between (n — l)th and nth renewal; but we assume that the pairs of 
random variables 


(Xs, Ys) n — 15 2... 


are independently and identically distributed. The s 
ofi i.d. random variables 


{Xn Y, п= 1, 2,... 


is said to generate a renewal reward process. 


equence of the pairs 


WU) 
Now Y()- 5 Y, 


a=1 
gives the total reward earned (or cost incurred) by time ¢ and 


(QY(0, t > 0} 
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is called a renewal reward process generated by {X,, Yn}. 
We have the following important theorem. 


Theorem 6.11. Suppose that 
E(X) = E(X,) and E(Y) = E(Y;) 
are finite. Then 
(a) with probability 1, Y(t)/1— E(Y)/E(X) as t— oo 
and (b) EY(t)/t—> Е(Ү)/Е(Х) as t œ. 
Proof of (a): We have 


t мМ) t 
О, 
N(t) t 
By the strong law of large numbers, with probability 1, 
NC) 
2 Y, 
п=1 
WO E(Y) as t-> со. 
and N(t)/t— Ip = VE(X) as t> oo (by Theorem 6.4). 


Hence we have the result. 
For proof of (b) reference may be made to some standard text, such 


as Ross (1970). A 


Denote 
EY) _¢ 
E(X) 

and Е(Ү(ї)) = C(t); 


C(t) is the average expected total reward (or cost) in [0, г]. Then from 


(b) above, we get, for large t, 
C(t) =. 
ed asymptotic form of C(t) has recently been given by 


A more refin t е 
ains the asymptotic expression for C(t) in the 


Christer (1978), who obt 


form 
C(-t:&T 5 


where Е and © are functions of the para 
the renewal process. 


meters and decision variables of 


renewal reward process. Consider a non- 
repairabie item with lifetime X, having distribution F. Suppose that 
replacements take place under an age replacement policy, with replace- 
ment interval Т, and that the costs of replacements upon failure and 
upon attainment of age T are A and B respectively, (B < 4). Then the 


Example 6(a). Age-based 
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associated age-based renewal reward process is given by {X,, Yn}, where 
Pr {Xn <x} = F(x), x<T 


= 1 x>T 
and Pr {Yn = A} = Pr (X, < T) — F(T) 
Pr {У, = B} = Pr {Xn > T} = 1 — F(T). 
T 
We have E(X) = E(Xn) = [ (1— F(x} dx 
0 
and E(Y) = Е(Ү,) =A F(T) + B(1 — F(T}. 
Then applying theorem 6.11, we have for large t, 
CO g Em 
бреу), 


In this case, the result 
C(t)/t = & (t > 0), 
can be obtained by straight forward calculation; the above provides an 


illustration of a reward renewal process. Christer (1978) gives an ex- 
pression of the term © in the more refined asymptotic form 


C(t) = t £ +s, 


and this may be used to obtain better approximation in the asymptotic 
case. 


6.7 Alternating (or Two-Stage) Renewal Process 


The renewal processes so far considered are concerned with the occur- 
rences of only one kind of event; the corresponding model relates to a 
system baving only one state. For example, we considered the working 
of a component, the lifetime (or time to failure) being given ‘by a sequence 
{Xn} of i.i.d. random variables, оп the assumption that the detection of 
failure and repair or replacement of the failed component take place 
instantaneously. Here the corresponding system has only one state—the 
working state and a renewal occurs at the termination of a working state 

: (or failure of a component). Consider now that the detection and repair 
or replacement of a failed item are not instantaneous and that the time 
taken to do so is a random variable. The system then has two states— 
the working state and the repair state (during which repair of the failed 
component or search for a new one is under way). Here the two sequences 
of states—the working states and the repair (failed) states alternate. 
Suppose that the durations of the working states (or lifetimes or times 
to failure) are given by a sequence of i.i.d. random variables and the 
durations of repair states (times taken to repair or search) are given by 
a sequence of ii.d. random variables. We have then an alternating or 
two-Stage renewal process. As can be easily imagined, a great many 
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renewal processes occurring in applications may be described by alter- 
nating renewal processes. . 

Consider an alternating renewal process. Suppose that the two alter- 
nating states are denoted by 1 and 0 and that the process starts at time 
t — 0 with state 1. Suppose that the duration of the two states 1 and 0 
are given by the sequence of i.i.d. random variables Y, (having a common 
distribution F,) and Y, (having a common distribution Fy) respectively. 
Assume that a renewal of state i occurs at each termination of state i. 
Suppose that X's and Y's are independent. Denote by N;(t), i = 1, 0, 
the number of renewals of state i in [0, 1]. Then {N,(t), t > 0} is an 
ordinary renewal process generated by the sequence of random variables 
(X; 4- Yi) having distribution H = +, (i.e. H is given by convolution 

. of Р, and Fo) and {N,(!), t > 0} is a moditied renewal process with 
initial distribution А, (і.е. initial interarrival time X,) and subsequent 
distribution H == Fj*F, (ie. subsequent interarrival times X; + Ү, 
Psed, 2 apd 

Denote the renewal functions by 

M(t) = ЕҲМ), i=1,0. " 


Let Л, fo be the p.d.f. of X; and Y; respectively and let Mj*(s), fi*(s) 
be the Laplace transforms of the corresponding functions. We imme- 
diately get (using (4.2) and (1.5) respectively) that 

A*S) 
M,*(s) = T 
iG = A) f) V 


* * 
MO = TERORO aa 


Theorem 6.12. For a system described by an alternating renewal process 

(starting with state 1 at £ = 0) the probability that the system will be in 
state i at time ¢ is given by 

p(t) = М1) — My(t) + 1 (7.3) 

and pot) = Муг) — M(t). (7.4) 

Proof: We first note that the | rth renewal epoch of the renewal process 

{No(t), t > 0) is given by 5, = - $ її + Yj. The distribution of S, is the 


r-fold convolution with itself nes = F,xF,. Thus the distribution of S, 


is given by H'* and has p.d f. h, (say). 
Now, the event that the system is in state | at time г is , equivalent to 


the following mutually exclusive events: 
(A): the duration of the initial state 1 (in which the system started at 


t = 0) exceeds t, the probability of this event being given by 
Pr (A) = Pr( > т} =1— F0); 


(B): in A и + du), for some u < f, a renewal epoch S, (for some 
r= l, 2, ...) occurred and the system remained in state 1 for the 


13 (45=95/1982) 
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remaining time (t — u). The event (B) has the probability 
Т 


Pr (8) = | [5 {Рг (и < S, < u- du} Pr (X, > t— u). 


[] 


The probability p,(r) of the event that the system is in state 1 at time t 
is given by 


fi 


POI- ROF | 5 һи) (1 — F, (t 0) du. (7.5) 
0 


Since L.T. of h,(u) is h,*(s) == [f,*(s) fs* (s), we have, taking Laplace 
transforms of (7.5), 
A 1 ж _ ise 
Pi (5) = E oO} : b OAOT Е ты ] 
ЕЎ) [ A) AP) 
5 (OG) GS) 


T САО 
Т тле) AS (7.6) 


From (7.1) and (7,2), we find that the above can also be expressed as 


pto = MEG — uto) i (7.7) 


from which we get, on inversion, 
P(t) = М0) — M(t) ; 1. 
It follows that 
Pot) = 1 — py(t) 
= M,(t)— M,(t). 
The theorem is thus proved. A 


Corollary. Steady State Distribution; 


Limiting Val, F 
Tf E(X), БУ) exist finitely, then Bate of PLD: 
lim Pt) -— 2) 
t>o E(X) -- Е(Ү) 
lim pr ЁЮ 
toe E(X): E(Yy 
Proof: Let 


E(X) - : a, E(Y) = b, Then 
A*G- 1—as: o(s) 


So*(s) -- 1 bs o(s). 
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Substituting in (7.6), we get 
as + o(s) 


BG) = рта 8) 5 +O} 


Now lim p,(t)= lim sp,*(s) 
>= 50 
un a 
“a+b 
and thus 
А ES) 
lim, PA = gy - EYY (7.8) 
It follows that 
i KY 
lim poll) = FEY) (7.9) 


Ix» 
of this alternating renewal process 


have led to interesting results. One such generalisation is that the 
system has Ñ states 1, 2,- -b «o o) such that transition from one 


State i to any other state / takes place with transition probability 


Note: Further generalisations 


N 
pu, X py =. 
ј=1 
е i before moving to state j is a random 


and that the time spent in stat 
ds on both these states. We have then 


variable whose distribution depen 

а semi-Markov process. 
Markov renewal proce 

renewal process. We deal with these two top. 


ss is another important generalisation of the 
ics in a separate chapter. 


6.8 Regenerative Stochastic Processes: Existence of Limits 
m 6.12, we were concerned with finding 
g distribution of p(t), as t— oo, i = 0, 1. 
Steady state distribution for stochastic processes in the more general 
Case is of considerable importance. Even without finding the exact 
limiting distribution, it is helpful to know about the existence of the 
limits in stochastic processes. Renewal theorem provides a very simple 
and powerful tool in proving the existence of limits or of steady state for 
a large class of stochastic processes. The result is given below as an 


existence theorem. 


In the Corollary to Theore 
steady-state behaviour or limitin 


Definition. Let (A(D) be à stochastic process having a countably 


ШЕПНЕ. number of sate lee жузше that there arc 
epochs 5, f, ... at which the process probabilistically restarts from 
= o 
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scratch, i.e. with probability one there exists an epoch /, such ня 
process be; ond t, is a probabilistic replica of the whole gi s UM 
at t= 0; then the same property holds for t,, t}, ... . Suc j pro 

is known as a regenerative process. Clearly {tn} forms a et process 
and every time a renewal occurs a cycle is said to be completed. 


Theorem 6.13. Let {X(t)} be a regenerative process and let 
pit) = Pr {X(t) = k). 
Then шй P(t) = px exists such that 


Pk 2.0 and 2 Pk d. 
Proof: Let ак) = Pr {X(t) =k, t, > t}. Then 
i 249 = = Рг {t, >t} =1— F(t), 
where F(-) is the d.f. of the interarrival time 


(fa — tna) = T, 
having mean, say, u. We have, by Conditioning on г, 
put) = | Pr (Xu) = k | һ = s} dF(s) 
0 
Li 


= qti | рк — x) dF(x); (8.1) 


since 961) == Pr {X(t) = k, >т} 


= [реро = => = заң) 


0 


= f Pr {X(t)=k | tj =з} dF(s). 
t 
From Theorem 6.3. we see that the renewal equation (8.1) has a unique 
Solution 
t 
Pat) = yt) + | aut ~ x) dM(x), 
0 


where M(x) is the ren 
distribution F, Now 


(8.2) 


ewal function of the 


renewal process with 
Gt) is non-negative, 


non-increasing and tends 


to 0 as t оо, and Í 902) dt < co. Thus application of the key renewal 
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theorem (Theorem 6.7) to (8.2) at once yields 


атайт 
im pt) > ИЛЗ | qi(x) dx, (8.3) 
0 
i.e. lim Pit) = px exists. Clearly, py > 0, 
о 
ic | ie г. 
E 5 m= Tal (E. at) de = t [ {1 — FQ} dx = 1. 
0 


Hence proved. A 
Note: Clearly, œ is the expected duration of a complete cycle, and 


Í q(t) dt = | Pr {X(t) = k, t, > t) dt == expected duration of sojourn 
0 0 

(or visit to) state k during a cycle = Е(хь), say. Thus 
Е(хк) 


Ё (8.4) 


py lim Pr{X(t)= k} = 
i-> 


Example 8 (a). Queueing process* with arrival and service rates à and p 
and A/u < 1. Let the state of the queueing process at epoch ¢ be given 
by the number in the system X(t) at that epoch; then the states of the 
process are i € E = {0, 1, 2,...}. Suppose that time is reckoned from 
the epoch 0 at which an arriving customer finds the system empty with 
a free server and the service begins, ic. with commencement of a busy 
period (with distribution, say, А). The busy period B is followed by an 
idle period / during which no customer is present in the system 
(I having distribution. say, Fy). A busy period and an idle period alter- 
nate and together constitute a busy cycle the whole—process restarts itself 
(probabilistically) from scratch at the end of each busy cycle. An idle 
period corresponds to the visit to state 0 while a busy period to states 


iz 0. Clearly, g(t) = 1 — ЕҚІ), 


а) dt = Expectation of the duration of idle period = E(1), 


and q(t) 4 = Expectation of the duration of busy period = E(B). 


0 
| 
Thus for such а system 

$ NOR E(1) 
p, = lim Pr (XW) —0 = дт) + Е) 


which is a very interesting result. 


*See Sec. 10.1 for description. 
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In this example .we were concerned with practically an alternating 
(two-stage) system; however the theorem is applicable to multi-stage 
systems also. 


Generalisation of the classical renewal theory: The classical theory (to 
which we devoted our attention) deals with a sequence of i.i.d. non-negative 
r.v.’s (Xj) and considers the asymptotic behaviour of EN) as t— оо 
(where N(t) = sup (n: S, & t}, 5, = i. ЕМО) = 2 Рг{5, < 10). 
Feller (1941) obtained the elementary renewal theorem which states that 
as t— oo, E{N(t)}/t—> l/u, where u= Е(Х). This result of classical 
theory was generalised step by step in several directions. We refer here 
to some of the works of the several researchers who made very significant 
contribution of immense mathematical and theoretical interest. This has 
been done to indicate the broad areas and to draw attention to some of 
the works for the benefit of those who might be interested in pursuing 
the topic more deeply. First generalisation was concerned with sequence 
of r.v.’s (Xj) which were not assumed to be non-negative, i.e. generalisation 
from distributions concentrated on the half-line to distributions on the 
whole line. Feller considers what he terms ‘transient distribution’ F on 
the whole line. 

Important generalisation of renewal theory to various cases of transient 
distributions, are due, among others to Chung et al. (Ann. Math. 55 
(1952) 1-6, Proc. Am. Math. Soc. 3 (1952) 303-309), Blackwell (Pacific 
J. Math. 3(1953) 315-320), Feller et al. (J. Math. Mech. 10 (1961) 
619-624). We state below (without proof) an important extension of 
Blackwell's renewal theorem. 

Let (Xi) be a sequence of (continuous) 1.1.4. random variables which 
assume values in the range ( — oo < x < co) such that 0cy—E(X)« oo. 
Let U(t) = I Pr(S, < t}, — оо < t< co, where $,— i Ху. Then for 
ASO, E ү 


U(t th-un-i as t+ œ 


Utt h) — U(t) 20 as t>—o., 


It is to be noted that in this case U(t) is, in general, not equal to E(N(t)), 
where M(t) = sup (n: S, < t} (for details of proof reference may also be 
паде to the books by Prabhu and Feller Vol. п). 

Attention was also turned towards the nature of (X). Chow and 
Robbins (Ann. Math. Statis. 34 (1963) 390-395) investigated the condi- 


tions on the joint distribution of X; which would ensure the validity of 
the elementary renewal theorem, 


X Pr(S,« Oft Mu 
n-0 


for some positive constant y. They relaxed the restriction of indepen- 


RENEWAL PROCESSES AND THEORY 199 


dence of the r.v.’s X;and obtained a renewal theorem for the dependent 
case as well as for a sequence of nonidentical r.v.’s. Smith (Pacific J. 
Math. 14 (1964) 673-699) also extended the theorem to non-identically 
distributed r.v.’s. Heyde (Ann. Math. Statis. 37 (1966) 699-710) conside- 
red asymptotic behaviour of more general forms like 


У a, Pr {Sa <t} and >a, Pr{ max Sk< t} 
n=0 n=0 IQkQn 
for a sequence of i.i.d. г. v.’s {Xi} and for a sequence of positive cons- 
tants (a,) such that Xa, diverges. 

Later in a series of papers Sankaranarayanan et al. (Pacific J. Math. 
30 (1969) 785-803, Act. Math. Aca. Sc. Hung. 31 (1978) 1-8, Math. Oper. 
Statist. Ser. Opt. 8 (1977) 581-590) considered suitable sequences of {an} 
and studied the asymptotic behaviour of such functions for sequences of 
correlated and dependent r.v.’s, as well as for stationary discrete 
stochastic processes. Renewal theory for Markov chains has been 
investigated by Kesten (Ann. Prob. З (1974) 355-387) and Athreya et al. 
(Ann. Prob. 6 (1978) 788-797) etc. Ryan Jr. (Ann. Prob. 4 (1976) 656-661) 
has considered the multidimensional case, while Chow et al. (Ann. Prob. 
7 (1979) 304-318) have developed an extended renewal theory. Thus 
extensions have been widely varied. 

The above is an illustrative (and by no means an exhaustive) account 
of the many interesting and far-reaching contributions made by several 
workers towards generalisation and extension of the renewal theory. 


EXERCISES 
6.1 Show that r 
ЖОО Es BU aD >». 


r=], 2,3,... 
17 
and Pr {N.(t)=0}=!— Al Pr {N(u) = 0) du. 
0° 
62 Taking f(t) g(t) as given in Example 4(a), obtain Mp(t) by using 
the relation (4.3). 


63 Show that the rene ion of a renewal process generated 


wal funct 


by Xn n=1,2,- ‚ is given by 
i р си? 
М) =: AC + о(1), 


and o? аге the mean and the variance of Xp. 


wh < oo) 
Mons mes that var (N(£) = (03u?) t + e(t). 
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6.4 


6.5 


6.6 


6.7 


6.8 


6.9 
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Find the second moment L(t) = E((N(1))*) for the renewal process 
with mixed expotential distribution having p.d.f. 
fit) = рае-® + (1 — p) be’, Oc p<l,a>b>0. 


Put forward direct probabilistic argument to show that, in the 
Markovian case, 


Pr (N(t) 2 n} = Pr {Na (t, Т) 2 п) = Pr (Ns (t, T) > n) 
for all t > 0, T20,n—0, 1, 2,.... 
Show that, for? 2 0, T > 0, п —0, 1, 2,... both (i) and (ii) 
G) Pr {N(t) > n) > Pr (Na (t, T) > n) 
(i) Pr {N(t) > n) > Pr (Ns (t, T) > n) 
hold iff F.is NBU, i.e. the lifetime X of the unit satisfies 
Pr (X > x + y) < Pr (X > x} Pr (Y > y) for x20,y20 
(Barlow and Proschan, 1975). 


Let R4(t, Т), Rg(t, T) be the number of removals in 10, :] under 
age and block replacement policies respectively. Show that 


Pr (N(t) 2 n) < Pr (R4 (t. T) 2 n) < Pr {Rz (t, T) > n) 
fort20, T20,n—0,1,2,.... 
(The intuitive meaning of the above is quite clear.) 
(Barlow and Proschan, 1964). 


Show that 
lim ЕКА (t. TY} e 1 
—>со 1 T » 


[o IRG) dr 
lim 204» (0 T) — M(t) +1 
1 Т 


1—5 


and that for all T > 0, 
MT) >F 
[ {1 — FQx)} dx 
0 


==] 


> Tju—1 (Barlow and Proschan, 1964). 
Consider the number № of renewals in random 
gamma distribution and p.d.f. 


time T' having 
ak th eat 

TO KE ae 

Show that the p.g.f. of N is given by 


G(z) = а t= us lil: 


q(t) = 
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6.10 


6.11 


6.12 
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Examine the truth of the following assertion: 
N has negative binomial distribution with p.g.f. 
aa rs 
G(z) Fak F = | 
if and only if the renewal process (X; is Poisson (ie. X; has 


exponential distribution with p.d.f., say f(t)= ce-*). 
Distribution of residual lifetime Y(t): alternative form. 


Show that m 
pro <= [t7 + w=) dM). 


If Z(t) is the spent lifetime at time ¢ for the process (X; and 


u = E(X) < ©, then, with probability 1 


a +0 as [+o 
and that 
E [29] as t— oo. 
Show that for the renewal reward process (X,, Yn}, the asymptotic 
form of C(t) is given by 


caters 
where Е = E(N) E(X) 
-- E(X? р 
апа t= EE zi ies E{Ynws} 
and that for the age-based renewal reward process considered in 
amply 2TE(X) — EX) EW) _ ACT — Е(Х)) 
{= |  XEQOF EQ). 


(Christer, 1978). 


Find M(t), = 1 0 for the alternating renewal process ТХЛ) 


with distribution of X, and Y, given by 
-ax, and F(x) - 1— €^ respectively. 


FQ) =1— ё 
Show that 
b а (arbre 
p(D-—zrb'adcb 
а _ 4 gay, 
pt) =} —plt) = гр b a D^ 
Further that "A. Жу 
lim p(0-— zb BX) + EQ) 
у EY) 


gud (5 " 
lim pit) = дв” EX) + El (Р) 
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Note that p,(f) for the Markovian case was obtained earlier by 
using results of Markov processes (see Example 5(b), Ch. 4), 
and that the result giving the limiting value of p,(r) as t — со 
holds independently of the form of the distributions of X and Y 
(see Corollary to Theorem 6.12). . 

6.14 Show that the number of renewals N(t) in [0, t] of a renewal 
process induced by X; is asymptotically normal with mean //y. and 
variance (o*/y5) t, v. and o? being the mean and the variance of X;. 

6.15 For block replacement (with replacement interval T) with costs 
of replacement upon failure and at intervals T, 2T, ... being A 
and В respectively, find E(Y(t))/t as t — оо, where Y(t) is the 
total cost incurred by time t > 0. 

6.16 Prove that the joint distribution of the residual lifetime Y(r) and 
the spent lifetime Z(t), is given by 
Pr {Z(t) > x, Y(t) > y} > b | (1— Аи) du 

PH as t -> œ (x > 0, у > 0) 
(where F denotes the distribution of the lifetime with mean y). 


REFERENCES 


A.A. Balkema and L. De Haan, “Residual lifetime at great age,” Ann. 
Prob., 2, 792-804 (1974). 

R.E. Barlow and F. Proschan, “Comparison of replacement policies 
and renewal theory implications,” Ann. Math. Stat., 35, 577-589 
(1964). 

——-—Statistical Theory of Reliability and Life Testing, Holt, Rinehart 
& Winston. New York (1975). 

A.H. Christer, “Refined asymptotic costs for renewal reward processes," 
J. Opns. Res. Soc., 29, 577-583 (1978). 

D.R. Cox, Renewal Theory, Methuen, London (1962). 

W. Feller, An Introduction to Probability Theory and its Applications, 
Vol. II, Wiley, New York (1966). 

U.S. Karmarkar, **Future costs of service contracts for consumer durable 
goods," AIIE Trans., 10, 380-387 (1978). 

N.U. Prabhu, Stochastic Processes, Macmillan, New York (1965). 

S.M. Ross, Applied Probability Models with Optimization Applications, 
Holden-Day, San Francisco, (1970). 

A.N. Sirjaev, Statistical Sequential Analysis, American Mathematical 
Society, Providence, R.I. (1973). 


W.L. Smith, *Renewal Theory and its Ramifications," J.R.S.S-B. 20, 
243-302 (1958). 


CHAPTER 7 


MARKOV RENEWAL AND SEMI-MARKOV 
PROCESSES 


7.1 Introduction 


Weshall consider here some kind of generalisation of a Markov process, 
as well as of arenewal process. We first consider a Markov process 
with discrete state space, and its generalisation by way of random trans- 
formation of the time scale of the process. Let X(t) be a Markov 
process with discrete state space and let the transitions occur at epochs 
(or instants of time) t, і = 0, 1,2, .... Then the sequence {Xn = X(tn + )) 
forms a Markov chain and the time intervals (/„+у — fn) = v, between 
transitions have exponential distributions, the parameter of which may 
depend on X,. We shall now suppose that the transitions {Х„} of the 
process X(t) still constitute a Markov chain, but that the transition 
intervals (tri, — tn) have any distribution, with parameter which may 
depend not only on X, but also on Х,а. The process (X,, tn} (as well 
as {Xm Tn} is then called a Markov renewal process and the process X(t) 
is known as a semi-Markov process. А semi-Markov process (s-M.P.) 
is thus a stochastic process in which changes of state occur according to 
a Markov chain and in which the time interval between two successive 
transitions is a random variable, whose distribution may depend on the 
state from which the transition takes place as well as on the state to 
which the next transition takes place. The s-M.P. provides a conve- 
nient way of describing the underlying Markov renewal process. Semi- 
Markov processes were first studied by Lévy and Smith, while Markov 
renewal processes by Pyke. Considerable interest in these topics has 
grown over the years, as, such generalisation of Markov process can be 
fruitfully applied in building models of many phenomena. Many prob- 
lems in queueing, reliability and inventory theories may be approached 
through Markov renewal and semi-Markov processes (see Cinlar 1969, 
1975 for a survey) In stochastic models for many types of social 
behaviour these processes háve been used (see Bartholomew, 1967). For 
example, problems of occupational mobility, where transitions 
between occupations as wellas intervals between transitions are taken 
into account, may be treated by means of semi-Markov processes. So 
also in some studies of problems relating to market behaviour. A semi- 
Markov model has been used by Kao (1974) in some context of hospital 
administration—in the study of the dynamics of movement of patients 
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i zones (e.g. medical unit, surgical unit, intensive 
t eid a hospital: E theory of birth has been developed as 
a Markov renewal process by Sheps (1964) (also Sheps et al., 1973), and 
the theory has been applied to microdemography by . Potter (1970) 
(see also Keyfitz, 1977). In fact, the importance of semi-Markov 
processes, in particular, and Markov renewal theory, in general, lies 
to a large extent in the wide domain of applications rather than in the 
richness of their theoretical developments. 


7.2 Definitions and Preliminary Results 


Definitions. Let the states of a process be denoted by the set 
E = (0, 1, 2,...), and let the transitions of the process occur at epochs 


(or instants of times) fọ = 0, f, ta.» -(fn < tn + 1). Let X, denote the 
transition occurring at epoch /,. If 


Prix inm faisse t| Хо... Ха 0571, way М}. 
= Pr (Ха =k; ъа — tn &t| Xn = -}, (2.1) 


then {X,, tn}, n = 0, 1, 2,... issaid to constitute a Markov renewal process 
with state space E. {Xn, Tn} is also used to denote the process {Х,, tn}. 
Assume that the process is temporally (or time) homogeneous, that is, 


Pr {Xn = К, tp — tn < t | X» = J} = Q(t) (2.2) 


is independent of n. Define 
pjk— lim Qjx(t) = Pr (X44 = k | Xn — j). (2.3) 
i-o 
(Symbols Q;, for Ож and p; for рук will also be used.) 


Proposition (1): 
{Xn n = 0, 1, 2,...} constitutes a Markov chain with state space E and 
t.p.m. Р = (pjk). ` 
The continuous parameter process Y(t) with state space E, defined by 
Қ) = Хь on th <t <tnyy (2.4) 
is called a semi-Markov process. The Markov chain {Xn} is said to be an 
embedded Markov chain of the semi-Markov process. X, refers to the 
state of the process at transition occurring at epoch t, and Y(t) that of 
the process at its most recent transition. Consider, for example, a simple 
random walk performed bya particle. Suppose that to ї,,..., are the 
epochs when transitions take place and that Y, is the transition at the 
epoch їл; also that at г, the particle makes а move to the 
enters the state j), at Ту to the left (enters the state /—1), 
right (enters the state j), at 1п+з tO the right (enters the st 
tnya moves to the right (enters the state j + 2), and so on. 


tight (and 
at tny. to the 
ate j+ 1), at 
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Then Y(t) =i, їз St< tnr 
S tni S 1 < tate 
=j, е, 


=j+1, tnta St < tna 


=jt2s thas St < fnis 


and so on. The position of the particle is shown in Fig. 7.1. 


Fig. 7.1 


Example 2(a). A pure birth process is а Markov renewal process with 
Ont) =1—e% k= s+! 

= 0, otherwise. 
t will be assumed in the following sections, 


Unless otherwise stated, i е assu 
rocess is finite having т states, 1, 2,..., m. 


that the state space of the р 


Waiting Times 
Let the time spent by 
that the next transition i 


distribution function (4. .): 
йд) = Pr (Tj & 0 — Pr (fni — fn & t | Xn Хн = k), 
„КЛ; 2, ул (229) 


the system in state j before its next transition, given 
s into state k, be a random variable Tj, having 


(called sojourn time or waiting time) depends on 
and the state Х to be entered in the very next 
f pjx = 0 for some j, k; then Qj«(t) = 0 for all 


The random variable Тук 
the state Y, being visited 
transition. Assume that i 
гапа Wj(r) = 1. Then 
W pdt) = QOP ik (2.6) 
conditional waiting time in state j given 


The variable Тук denotes the e 
f T; denotes the unconditional waiting time 


that the next state is k; and i 
in state j, then ; 
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Tj = EpsTy (2.7) 
and W(t) = Pr {T; < t} = z Dik Wt) = Qt). (2.8) 
We also have 
Mf) = Е(Ту) = z рук E(T},) =E pyk afp, 
r=1,2,... 


whenever the moments of? of order r of Ту exists. Denote ма by М; 
апі of) by «ук. We have 


ajk = | Pr {Ti > t) dt = | {1 — Wt) dt (2.9) 


ormi 


and М;= 


om} 2—3 


Pr (T; >t} а= [а = È OnO) dr; (2.10) 


М ;іѕ the expected amount of time spent in state j during each visit and 
a jx is the (conditional) expected time spent in state j during each visit 
given that the next state visited is k. 

Note: If Oj(t) is of the form 


Qjx(t) = рук (1 — е9 "} 


(equivalently if W(t) has the form of the second factor), then it can be 
shown that 


Pr (Y(t-- s) = k | Y(u), u < t) — Pr Qt + s) = К | Y(t} 

and that 
Pr (Y(t + s) = k | Yt) =j} = рум) 

is independent of t. In this case the semi-Markov process becomes a 

time-homogeneous Markov process with t.p.m. P-—(pjj). (In fact, 

this is what we get in Example 2(b) given below.) The Markov property 

which holds, in case of semi-Markov Processes only at the epochs їл, will 


hold for all t only when Озі) is of the above form. Thus in semi- 
Markov processes there is freedom in the distributi 


times Тук, but only at the expense of the Markov pro 
of holding for all t holds only at the jump epochs fp. 


A Markov process is a semi-Markov process but the converse is 
not true. 


Suppose that X(t) is a s-M.P. with discrete time, 


intervals (tn; — 1,) are equal to unity for all 
{Xn}, where X, = X(t,-- 


on of the sojourn 
perty, which instead 


and that the transition 


n. Then the sequence 
) reduces to a simple Markov chain. 


Example 2(b. The M/M/1 queue*. 


Suppose that customers arrive at 
a counter in accordance with a Poisson 


process with parameter à and are 


*For description, see Ch. 10. 
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served by a single server, the service times of customers being i.i.d. 
exponential random variables with mean I/p. Then {N(t), t 2 0}, where 
N(t) is the number of customers in the system at time t, is a s-M P. (it 
is also a Markov process). The states of the s-M.P. are 0, 1, 2, .... 
Suppose that fp, f, ... are the successive instants at which changes of 
states (by way of arrival or departure of a unit) take place and 
X, = N(ts + 0); * 
Qo,a(t) = Pr {Хы == ls tpa — tn < t| Xn = 0) 
== Pr {the process enters state 1 from state 0 by time t} 
=l-e%, 
О) = Pr {Xn =] l tata tn & t| Xu =f}, 72> 1 
= Рг (the process enters state / — І from staie j by 
time t} 
= Pr {опе transition (service completion or arrival) 
within time £}: Pr {transition is service completion) 
= Pr (min (A, B) < t)- Pr (4 > В), 


where the random variables A, В denote respectively the interarrival and 
service times (which are independent exponential with mean 1/Х and 1/y). 
Since Pr { A > В} = шо + ш) and min (4. B) is exponential with mean 


1/(A + p), we have 
Oj O= (1 — ey QUO p) Jj» 1. 


Similarly дым) — Pr (min (A, B) < t)-Pr {B > A) 
= (1 e09m-Q/Q t gj 
Ом) = 0, fork j— bj 1O k > 0. 


Further 
Hence the transition probabilities of the embedded Markov chain (X;) are 
Poi = 1, p= k#j-1,j+1 
Prj- = BIA + s Phm = NA+ wj 2 1. 
Thus W,s(t) = Pr {ы — fn €! | Xn =0, Хы = 1) 
epe 


Ора №100), J 2 1 
3j- e 
Further, Т,, the unconditional waiting time in state 0 is the same as. the 
interarrival time, ie. To is exponential with mean 15. T2 1), the 
Unconditional waiting time in state / is the time required for a transition 
(either an arrival or service completion). Thus T; which is equivalent 


to min (4, B), is exponential with mean 110 +p) We have 
MEE ET) = ІА 
м, = ЕТ) = M0 + uw), J > 1. 
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Example 2(c). The M/G/1 queue. Here we suppose -that the service 
time distribution is general, other things being the same as in the M/M/1 
queue. We note that (N(r), t > 0} isnot asemi-Markov process, because 
the state of the system after a transition depends not only on the state of 
N(t), but also on the amount of elapsed service time of the Person receiving 
service, if any. If we consider that transitions take place only when a 
customer completes his service, there will be no elapsed servicetime. X, 
will here give the number of customers in the system at the instant that nth 
customer departs. If Y(t) denotes the number of customers left-behind 
in the system by the most recent departure, Y(t) = Xn, t, < t< [m 
and Y(t) will be a semi-Markov process having X; n-— 0, 1, ... for its 
embedded Markov chain. (X,, t5), n —0, 1, 2, ... isa Markov renewal 
process. The sequence of intervals (у — їл), n = 0, 1, 2, ... being the 
service times of successive units (or equivalently (tj), n = 0, 1,2, ...) 
forms a renewal Process (as explained below). 

We now examine how Markov renewal processes can be viewed as a 
generalisation of renewal process. It can be shown that, for n = 1:2. > 


Pr {t-t < Hs fo — ty $ Uz ..., fn— In 4 S Un 
dyes X; mE x in} 
= Wai, (иу) Иш» (из) ... Ий (un). (2.11) 


In particular, if there is only one state, then the transition intervals 
(fa — 1,4) are 1.1.4. non-negative random variables. 


Proposition (2): 


If E consists of a single point, then the Sequence {tn}, n= 0, 1, 2, ... 
constitutes a renewal Process. Suppose j € E is fixed andi Sisi ce 
are the successive epochs t, for which Xn =j. Theü for each fixed Jj. the 
sequence {Sj}, n -- 0, 1,2, ... constitutes a renewal process which is 
ordinary if j is the initial state (with S,/ = T, = 0) and delayed, otherwise. 
Proposition (3): 


To each state j € E, there corresponds (a possibly delayed) renewal 
process (S2), j == 1, 2, ..., m and the superposition of all these renewal 
Processes gives the set of transition epochs fs, f, tf, ... . The fact that 
{Xn} constitues a Markov chain and the fact that {Si}, j € E constitutes 
renewal processes provide a justification for the terminology Markov 
renewal Process—as a generalisation of Markov chains and renewal 


processes. In view of proposition (3), an alternative definition of the 
Markov renewal process is also given below. 


Definitions. If N;(t) denotes the nu 


mber of transitions into state j 
(renewals of state j) in [0, ¢], then 


Ме) = {N (0), №0), ..., Nt), 
is called a Markov renewal process. 
Suppose that the process starts with state / at time 0. 


(2.12) 


y i6. Жуз XO) St 
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Define ў 
G;j(t) = Pr (N;(t) > 1| Y(00) =i} (2.13) 
AD = ш a Noh 2.14 
OD = P Qo min e rise 2719 


Gy,(t) is the d.f. of the first passage time Vi; into state j given that the 
initial state is i. Неге irrespective of the number of transitions 
(into other states) involved, the visit to j must be for the first time. Gy), 
which gives the d.f. of the time beiween two occurrences of the state j, 
may be different from G;; (i+ j). Denote E(V;j) = ш, when it exists. 
A more detailed discussion on first passage time is given in a latter 
section. 


Now 000 (1) = 1 or 0 depending on j= i or not. Denote 00 by Q. 


Proposition (4): 
For n=0, 1,2, ... 


СТЕ (2.16) 


This can be seen by considering that of the (n + 1) transitions, the first 
one takes place at some time u ( < t) and reaches some state К, and the 
subsequent п transitions from state k take place within time t — u, and 
end with j. 
Markov renewal function. The functions 
Mii, j, t) = E {N,(t) | Xo =i}, je E (2.17) 
are known as Markov renewal functions corresponding to О. 
Proposition (5): We have 
M, у= X Qi? (0). (2.18) 
This can be seen by noting that , 
ч М) = Z An, where 
An=1, if X =j, t, «t 
— 0, otherwise 
and Pr {4,=1|X%) =i} = Pr {Xn =j, th t| Xy i) 
= Q(t). (2.19) 


For j = i, the interarrival times between two occurrences of i (having d.f. 


Gu(t)) induce an ordinary renewal process. Denoting the п fold con- 
volution of Gy with itself by бу, we can write the Markov renewal 


14(45-95/1982) 
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function (i.e. conditional expectation of the number of entrances to 
state i) as 


M(, i, t) = X Gr. (2.20) 


Proposition (6): | 

Blackwell’s and Smith’s renewal theorems hold for the (ordinary) 
Markov renewal function M(i, i, t). 

Classification of states, Suppose that the initial state in a Markov 
renewal process (X, tn} is i. The state i is said to be recurrent (transient) 
iff Gu (со) == 1(« 1). A recurrent state is positive or null recurrent de- 
pending on uj; <,ог = œ. We have the following result. 


Proposition (7): 
For the Markov renewal process (X, tn} state i is recurrent (transient) 
iff state i is recurrent (transient) in the embedded Markov chain {Xn}. 


7.3 Markov Renewal Equation 


We obtain a Markov renewal equation which gives a relationship 
between f;; and Q;;. 
Theorem 7.1 For all i, and for t 20, 


fi) = y MD E | fut — х) dono, Q.1) 
LJ 


where hr) =1— Z Qu(t) — 1 — Wt) = Pr (T; > 0). 


Proof: We apply renewal argument 


2 › by conditioning on the time f, 
of the first transition as well as the state t 


hen entered, we get 
fit) = Pc (Y(t) «j| X(0) = i) 


ES I J Pr (Y(0) =j | Y, =i, A =k. ty = x} dQj(x). 
0 


If t, — x 5 t, ie. the first transition epoch is be 
=i If t, — x < t, then starting from state k (г 


by time х) ће process ends up with state j by th 
Thus 


yond г, then Y(t) = Y(0) 
eached at first transition 
e remaining time (t— x). 


Pr(Y( — j| x,— i, =k, = х} = 8, ifx >t 


Непсе = Set х), if x « t. 


ло ® | EAEN Í 40.) 
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Since zÍ dos) = 1, 
k 
0 
the second term 


= ð; (1 — 2 J 4Ок(х)} 


=o 2 Qu(t)) = 8,0) (3.2) 


and thus we get (3.1). A 

The equation (3.1) in fi;(t) can be solved as a Markov renewal equa- 
tion. We shall not consider this aspect here. We are interested mainly 
in the asymptotic behaviour of the stochastic process Y(t), which will be 
found without explicitly finding the solution f;;(t) first. We shall intro- 
duce a matrix similar to the t. p. m. of a Markov chain. 


Interval Transition Probability Matrix (i.t.p.m ) 

The matrix with functions of /, k;,(t), i j= 1, 2,..., m as elements will 
be denoted by КЦ), i.e. K(t) = (k;(f). The matrices-obtained by taking 
derivatives, integrals, Laplace transforms etc. (provided they exist) of the 
elements will be represented accordingly. For example, 

t 1 
[ K(x)dx = ( Í Кх) dx) 
0 0 


K*(s) = (kj (5), 


where k* (.) denotes Laplace transform. 
` We shall refer to these matrices here 


F(t) = Cfi). 

C(t) = (cift), where c(t) dt = pd Wit) = dQ;((t) 

A(t) = (ai(0), where а; (1) = 8); (1). (3.3) 
The matrix A isa diagonal matrix. Since fiı;(t) 2 0 and > f(t) = 1 for 
all ¿= 1,2, ..., т, the matrix F(t) is a stochastic matrix; it is called 


interval transition probability matrix (i.t.p.m.). 
Now (3.1) which holds for all states i, j= 1, 2, ..., m can be put as 


fit) = а) + [ z en(x) fe(t х) dx, і) =1,2,...,m (3.4) 


0 


and in matrix form as 
t 
F(t) = A(t) + [ CE — х) dx. (3.5) 
0 


Denoting the convolution of C and F by C*F, we can put the same as 
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F(-) =A(-) + C(- EC). (3.6) 
Taking Laplace transforms, we get 
F*(s) = A*(s) + C*(s) F*(s). (3.7) 


Note that C is a somewhat special type of matrix. 
7.4 Limiting Behaviour 


Asymptotic behaviour of a stochastic process is a question of great: 
general interest. We shall now investigate such behaviour in case of a 
semi-Markov process. Suppose that (X,, tn} is a Markov renewal process 
and Y(t) = Xj, tn < t < Га, is the semi-Markov process induced by Q(t). 
The limiting distribution of the embedded Markov chain {X,} is not the 
same as that of the semi-Markov process. If the Markov chain is irredu- 
cible and ergodic and has the t.p.m. P = (p;;), then the limiting 
probabilities lim р =v; јЄ Е are given as unique solutions of 
у= 2 урь, је E, i.e. the probability vector V = (v,, v, ...) is the 
unique solution of V — VP. 

In the ergodic theorem given below, the limiting distribution 


lim 700) lim Pr(Y() =/| Y) =i} 


is obtained. 


It also gives the relationship between the two limiting 
distributions. 


Theorem 7.2 Suppose that {Xn} is irreducible and ergodic and that the 
mean waiting time in state i, M; < co for any i € E, then 


vM; і 
М (4.1) 
k 


fim fi) =fiy = 


Proof : Suppose that the variables Туу are continuous having p.d.f. 
wij (f). Then T; із also continuous and has the p.d.f. wir) = Х pj w(t). 
Referring to (3.3), we get ў 

i l wi 
ch (8) = pu wh ($), ay (s) = 80) hë (з) = ij 5 — ug. (4.2) 
Taking limits as s+ 0, 
d А 
сў (s) > риу, аў (5) > 81 ds C- wf) l= = 80M, (4.3) 
so that as s > 0 І 


C*(s) = (cis) > (pi) = P (4.4) 


А*(5) = (aj(s)) > (81; Mi) = М; 


М is а diagonal matrix with diagonal elements M;, My hiss 
From (3 7) we get 


and 
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Ur — CAFS) = AX), 
I being the unit matrix. 


Thus sF*(s) = B(s) A*(s), (4.5) 
where B(s) = s [I — C*(s))?. (4.6) 
Hence Bis) — C*(s)] = 51 
or B(s) — B(s)C*(s) = s1. 

Denote lim B(s) = В = (01); 
then taking limits as s— 0, we get 
B—BP=0 
or B= ВР. (4.7) 


Thus we find that B satisfies the same equation as V, i.e. 


by = варь) = 1, 2, ... (4.8) 


y; = Укрк» ј= 1,2, ..-- (4.9) 
k 


The solutions of (4.9) ате unique. As both the sets (4.8) and (4.9) 
hold, the rows of B and ү must be proportional, i.e. 


bij = йу» where d, are constants. 


Now fi; = lim ЛК) = lim sfu*G)s 


t-> 
writing 


F= (fu) = lim (fi) 
= lim (sfis*(S)) = lim sF*(s), 
50 50 


we have from (4.5) d 
F = lim sF*(s) = lim B(s) lim A*(s) 


50 
— BM. (4.10) 
Thus fi” bjM;— div; Mj. 
; = M; for all i, 
Since xev dico j lor a 


finally we get 
y we g йл, 
lim fu) =u = Sy, Me 
too k 
The limits fj(f) are independent of the initial state i; we may write 
fij — f. 

To complete t 
ables are discrete, by cons! 
transforms, and proceeding ОП 
result. A 


he proof of the theorem, we note that in.case the vari- 
oe idering generating functions instead of Laplace 
the same lines, we can obtain the 
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Particular case. When the waiting times are all exponential and 
Мт) = 1 — e-^' , then Y(t) becomes a Markov process. | We then get 
M;=1/a;. Substituting this value of M; in the expression for fj one 
can ultimately arrive at the expression (Eq. (5.19), Ch. 4). 


Z fax = 0, (2 f, = 1) 
J 7 


satisfied by the limiting distribution (fj) of an irreducible Markov 
process 


When the semi-Markov process reduces to an ordinary Markov chain, 
then M; = 1 for all jand one gets f; = v,,as is to be expected. The 
relation fi; = у; holds also in the case when the unconditional waiting 
times in all the states (i.e. interoccurrence times between recurrence 
points) are independently distributed with the same mean M, — M for 
allj. It may be noted that the expression (4.1) giving fj, involves only 
the v/'s, the limiting transition probabilities of the embedded Markov 
chain and the mean waiting times M; == Pik «ук, ie. the mean су and 


no other statistic of Тук. The distinction between the quantities v; and 
J; must be carefully observed. Suppose that a process has been in opera- 
tion for a sufficiently long time and that a sufficiently large number of 
transitions have taken place so that a Steady state has been reached. 
Suppose that the situation at some instant is considered. The quantity f; 
gives the probability that the process is in (or occupies) the state j, while 
Y; gives the probability that, given that the Process is now making a 
transition, the same is to state j. Except in the cases Stated above the 
two quantities v; f; are unequal. Even in the case when the process is 
a Markov process in continuous time they are unequal (see Example 4(b)). 
We consider a numerical example below. 


Example 4(a). Suppose that the embedded Markov chain {Xu} has 
the t.p.m. 


0.5 0.3 0.2 
P=|0.2 0.4 0.4 
0.1 0.5 0.4 


and that the matrix of complementary distribution functions of Туу is 


e e 5t e 
W =| е-и et е-ы |. 
е5: et e 


Using M; = Z ру, we get 
i 


М; — 0.51, M, = 0.33, М, = 0.67, 
the limiting probabilities уу = lim p(? have values 
n> Д 
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y, = 0.2353, v, = 0.4118, v, = 0.3529 
(see Example 4(b) Ch. 3). 
Using (4.1), we get 
f= 0.2437, у, = 0.2760, f, = 0.4803. 


Example 4(b. Consider the М/М/І queue discussed in Example 2(b). 
Suppose? < p, and denote 
a=p/A+ u), b = МО + и); 
then 
Pon = 1, Pj) = % Ры+ = b,j > l, pe=0,k#j—-1,j+1. 
The elements of V = (Y, у, Y» ---)» 2 vı = 1 are the solutions of V = VP. 
We have 


» = Vy 
у, = Vo + 4 
and ук = Бук-у-_ укр К 2,3, .... 


The solution of the last equation is given by 
vk = c, + c (b/a)*, k = 1, 2, .... 


Solving in terms of vy, we get 
c, = 0 and c, = vob, 
ve = (1/6) (6/a)* vo 


= (ay, E 
-( - ) 2) Vee; ди. 


Now My = 1/\, Me = 1/(% +u), K=1, 2, ... 
so that 
= Vo 
rele = т My 
Thus we get 
vM; 


This result obtained earlier, as solution of difference equations, is consi- 
dered here as an illustration of application of the theory of s-M.P. 


Note that Vk > fk, k > 1, vy < fo 
Limiting Distribution of s-M.P. and ‘Recurrence Times’ 


There are many other distributions connected with Markov renewal 
processes, whose asymptotic behaviour is of considerable significance 
and many limit theorems have been obtained. We shall discuss here the 
limiting behaviour in another important case. 
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Suppose that (X,, tn} is a Markov renewal process and that 20 is its 
associated s-M.P. For each f, there exists л, such that Inst < бз; then 
U(t) = t — tn, V(t) = tnt: — t are respectively called the time since the last 
Vc and the time until the next transition (‘recurrence times’). = 
applying renewal type of arguments and using the renewal theorems, t И 
limiting form of the joint distribution of Y(t), U(t), V(t) can be obtaine 
as stated below. For details of proof, refer to Pyke (1961), Cinlar (1969) 
etc. 


‘Theorem 7.3. If the process is irreducible and not lattice, then, as 
t— со 


Рг{Ү(@) =}, Xn = k, U(t) 2 x, V(t) > у | YQ) = i} 


05 l Pil — Wo) du, (4.11) 
r ч { 
provided the integral exists. 
Summing over k, we have, as г> со 


Рг (Уб) =j, U(t) > х, Vt) >y] YO) i 


^y "in [a = 2 Qix(u)} du =F | Pr {Tj >u} du. (4.12) 
r x+y x+y 
Thu, Pr {U(t)>x, V(t) >y j Y(t) ==}, Ү(0) =i} 
E Pr (T; >u} 4.122) 
EM f ALIE du. (4. 
x+y 


Putting x=0 in (4.11), we have, as г> oo, 


Pr (Y(1) =j, Жыз == К, V(t) > у | Ү(0) = i} 
Y; 
MIU | Pik{l — Wix(u)} du. (4.13) 
r P. 
It may be noted that when the 
ordinary renewal process 
yields the limiting distri 
gives the limiting joint di 


Markov renewal process reduces to an 
(i.e. there is only one state), the above (4.13) 
bution of the residual lifetime, whereas (4. 12a) 
stribution of residual and spent lifetimes. 


7.5 First Passage Time 


We consider here some of their properties. 


First, we obtain a relation- 
ship between С; (1) and At). 


Theorem 7.4. For all i, j and for t > 0, 
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t 
fu) = Bij (+ | fuc х) або), 5.0 
0 
where A(t) = Рг (Ti t). 
Proof: Theevent Y(t) =j, given Ү(0) = i can happen in the following 
two mutually exclusive ways, which give the first and the second term 
respectively on the r.h.s. of (5.1): - 


(i) No transition occurs by time t; then Y(t) = i, and the time spent 


in state i exceeds t. 
(ii) One or more transitions occur by time t. Suppose that the process 
e first time by time х < f and then starting from 


reaches the state j for th 
this initial state j, the process in the remaining time г — x, ends up again 


with state j. A 
We now obtain a relation between Gi j(t) and Q;;(t). 


Theorem 7.5. For all states i, j and t > 0 


t 


бй = Ou) + 2, | Gu — 9 409 (5.2) 
0 
= ODHE [ Gut — э) ао) 
0 


= | 602 — х) 4015). (5.2а) 
0 


Proof: The first term on the r.h.s of (5.2) corresponds to the case 
of direct transition from state i to state j in the first transition within t; 
the second term corresponds to the case of transition from state i to k 
(52 Ј) in the first transition at time x < t and then starting from the in- 
itial state К, the process, in the remaining time (t — x), ending up with 


statej. A 


Laplace transform formulation. 
continuous and 


Suppose that the variables Уу, Tj; are 


dG; (x) = gij(x)ax (5.3) 
а0, (х) = pi; айх) = ру w) dx, 
Then taking Laplace transforms of (5. 1), we get 
70 = Bij №5) + 8005) £5), 
HMEE 
so that Ji) = EO 0) =i (5.4) 
"T 
rape) = SAO, ju (5.48) 


1 — 3) 
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Taking Laplace transform of (5.2), we get 


Сз) = Ор) + X GG). pu 
Рашн» B= AHO) + ® seo. (5.0) 


Denoting (8;,(5)) = G(s) and the diagonal matrix ugl) = G;*(s), 
(5.6) can be written in matrix form as 


G*(s) = C*(s) + C*(s) G*(s) — C*(s) G;*(s). (2308) 


' Mean first passage time. 1f С, (оо) = 1, then 
i = EV) = f 146,0) (5.7) 
0 


is the mean first passage time and ujj is the mean rec 


urrrence time of 
state j. 


Theorem 7.6. For all i, j 


—Miz : 5.8) 
pu ! куё) Инн) ( 
Proof: Suppose that the variables у, Туу are continuous and that 
the relations (5.3) hold; then 2100) = w#(0) = 1 


d 
Hy = — EI) sao 


d 
91у = — АУ ӨЛЕ 


and Mi= 2 Pik us. 


From (5.6) we get 
d d d d 
609) = pu WU) + 2. Vou wi) 4 куу + р едг) FIO 


Hence 84) = Pij oy + PA шо + Pik aig} 


= У рьш + X 
x ik ah Ш 


= М, > А 
i+ "m Ше 


The proof, when the variable: 
manner by considering the g 
their Laplace transforms. 


Note: (1) The intuitive m 


passage time from state i to Jisthe meantime required 
state į plus the sum of the product of meanti 
from state k (for all k except k 


probability that the first transiti 


Sare discrete can be given in the same 
enerating functions of Vi;, Ti; in place of 


eaning of (5.8) is clear. The mean first 


to leave the 
Me required to go to state j 
=j) reached at first transition and tbe 
on is from state ; to К. 
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(2) Higher moments of Vi; can be obtained from (5.6). We obtain 
another interesting result. 


Theorem 7.7. If the embedded chain (X5) is irreducible and ergodic, 
then for all i, j 
ey = E Mir). (5.9) 


Proof: Since {Xn} is irreducible ergodic, v, are given by unique 


solutions of ; 
у = 2 эрк 


Now multiplying both sides of (5.8) by v, i = 1, 2,... and adding, 


we get 
z nu = и М, + re vipi Uk; — "iij jl 
=} nM, + Х {> ир} Bay — È ири} eji 
= z viMi + I Уке — VR 
Thus uy zn wMiv,. А 


Note: We сап write (5.9) as follows: 
ш = = Mil" )- 
e meaning of this expression would be clear once it is 


ed number of visits to i between two visits 
time whose mean is Мі. 


The intuitiv 
noted that (vi/v;) is the expect 
of j and each visit to i lasts a 


When the s-M.P. reduces to a simple Markov chain, then 


Corollary 1. 
d then we get the well known result 


M; —1 for all j an 
jj = Шу). 


Corollary 2. From (4.1) and (5.9), we get 
f; = Муш» (5.10) 
which gives fj without directly involving v/'s. 

It can be easily seen that the results derived for a finite set E will also 
hold good, in general, even for a set E containing a countably infinite 
number of points. 

We have covered, within the scope of this book, the most important 
aspects of Markov renewal and semi-Markov processes. The niceties of 
theoretical structures and the vast potentialities of applications have 
attracted a large number of researchers to these topics. The literature 
has grown tremendously over the last two decades, from about 5 titles in 
1960 to over 400 now. It is not possible to list them all here and reference 
may be made to Cinlar (1975) for a list of 70 titles and to the mimeo- 
graphed compilation by Cheong et al. (1974) for an exhaustive list. 
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7.3 
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EXERCISES 


The embedded Markov chain of a two-state (1, 2} s-M.P. has 


t.p.m. 
2/3: 01/5 
ps 
12 12 
and the matrix of the complementary distribution functions of 
the waiting times is 


et e-3t 
иа РУ ж a) 
Find (i) the L.T.’s of interval transition probabilities f(t), 
Jr i12 
(ii) the limiting Probabilities Peel iD) 
For Problem 1, find 
() the first passage time densities g(t), 


(ii) the mean recurrence times of the states 1, 2, and 

(iii) the mean of the first passage times from state j to state 
K,j#k. 

Suppose that the employment status of certain section of labour 

Population in an area can be described by a Markov chain with 

two states employed (0) and unemployed (1) and that the chain 


has t.p.m. 
(к 0.6 
P = . 
0.7 0.3 
ting times densities аге given by w,,(t) = ет, 
*, wolt) = te~ and walt) = 312 ert, 


St passage time? 

(iii) What is the limiting Probability that he will 
and (b) unemployed after a s 

Entrance Probabilities. L, 

the process will enter sta 

it entered state ; аба 


be: (а) employed 
ufficiently long time? 
et e(t) dt denote the Probability that 


te j in the interval (t, t+ dt) given that 
0. Show that 


t 


fult) = | euf) hit — x) dx, 
0 

һи) = Pr {Т) > u}. 

If the process is irreducibl 


where 


* and ergodic, show that lim ej(f) = e; 
too 
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7.6 


7.8 


7.10 


7.11 


7.12 


exists and is independent of the initial state 7; the limit е, (called 
the limiting entrance rate to state /) is given by 


e; = f) M; 
and further that e; = Ши; 
Show that the second moments 


EV$)-ug— [ 1? dG;,(t) 
0 


of the first passage times Vj; (with d.f. Су) satisfy the relation 
up =MO+ E ри (2ле) + ug 


Use this relation to find var (V3) and var (V,,) for Exercise 7.1. 
Show that for an irreducible and ergodic process, if ;N; denotes 
the number of visits to state j іп between two visits to state j, 
then 

E (Ni) = vil vy 
also E (Ni) = шуш. 
Show that for an irreducible ergodic process 

. M(k, j, t) v; 

кам») w 
For the M/M/1 queue (exponential interarrival and service times 
having mean 1/а and 1/b respectively) and with batch service 
(taking in a batch not more than с units) find the expressions 
for Qij(1) and pij. 
For the M/G/1 queue (with exponential interarrival time having 
mean l/a and general service time with distribution B(x) and 
mean 1/b) taking tn as the epoch of the nth departure and Y(t) as 
the number left in the system by the most recent departure, find 
expressions for Q;,(t)- 
Repeat the above exercise with the modification that service is in 
batches of size not more than c (i.e. of size c or the number in 
the queue at the time of initiation of service, whichever is less). 
Examine the validity of the relation f; = v; in case of GI/M/l and 


M/G/\ queues. 
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CHAPTER 8 


STATIONARY PROCESSES AND 
TIME SERIES 


8.1 Introduction 


A series of observations x(t) made sequentially in time / constitutes a 
time series. Examples of data taken over a period of time are found in 
abundance in diverse fields such as meteorology, geophysics, biophysics, 
economics, commerce, communication engineering etc. Daily records of 
rainfall data, prices of a commodity etc. constitute time series. The 
variate t denotes time, і.е. changes occur in time. But this need not 
always be so. For example, the records of measurements of the diameter 
of a nylon fibre along its length (distance) t also give a time series. Here 
t denotes length. 

The essential fact which distinguishes time series data from other 
statistical data is the specific order in which observations are taken. 
While observations from areas other than time series are statistically 
independent, the successive observations from a time series may be 
dependent, the dependence based on the position of the observation in 
the series. The time / may be a continuous or discrete variable. A 
general mathematical model of the time series Y(t), t &- T is as follows: 


Y(t) =f) + X). (1.1) 


Here f(t) represents the systematic part and X(t) represents the random 
part. These two components are also known as signal and noise respec- 
tively. The model is theoretical; f(t) and X(t) are not separately 
observable. While the model for Y(t) gives the structure of the generating 
process, a set of observations (or time series data) is a realisation or a 
sample function of the process. The effect of time may be in both the 
systematic and the random parts. 

A general model in which the effect of time is represented in the 
random part involves a stochastic process, і.е. a random function of 
time. The systematic part is represented by a (non-random or deter- 
ministic) function of time. "The important types of evolution (in time) of 
the systematic part are described by the following components. 

(i) Trend: It is a slowly varying function of time due to growth 
and other such permanent effects. A trend is usually represented by 


means of a polynomial of a low degree or a short Fourier series. 
(ii) Cyclical component: Tt is a broad long-run movement, more or 
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less regular, caused by fluctuations in activity and is quasi-periodic in 
appearance. ; "€ 

(iii) Seasonal component: It is composed of regular variations, 
caused by changing seasons, rhythmic activities etc. Cyclic and seasonal 
movements are usually represented by suitable periodic functions. 
Trigonometric functions are generally used for representation of cyclic 
movements. Both the systematic and random parts may involve a 
number of unknown parameters. The two parts are considered to be 
additive. 

For description and analysis of components involving the systematic 
part the reader may consult the excellent standard texts in time series, 
such as Kendall, Kendall and Stuart, Malinvaud, Anderson, Fuller etc. 

We shall discuss here only the models, where the random part involves 
stationary stochastic processes (real-valued), Here we shall also confine 
mostly to processes in discrete time, A time series {xs £— 1, ..., m) may 
be considered as a single particular realisation of the stationary stochastic 
process (X, t & I). We shall be concerned here with the stationary 
process (X, t & I} rather than with- the analysis of the time series 
{xs t—1,...,m). 

As discussed in Ch. 2, a Stochastic process {Ynt € I}is said to be 
stationary in the wide-sense or covariance stationary iff, for all гє J, the 
mean value function E{X;} = m is independent of t and the covariance 
function E (X, — m)(X,,, — т)} = Ck is a function of the difference 
(їп lag) (Y+ К)—)=К. The correlation function is рк == Ck/Co, where 
Co = var (Х,). The graph of the correlation function is called the 
correlogram of the stochastic process. 

Some examples of stationary processes were given in Ch. 2. We shall 


consider below some stationary processes which are used as models for 
the generation of the random part of a time series. 


8.2 Models of Time Series 
(a) Purely Random Process (or white noise process) 
A completely random process (X;) has 
E(X) = m, Q.D 


Which may be assumed to be 0, for simplicity, and 


EX, Ху = {Ok =0 
At Xia} t ka 0. Q.2 
The process is covariance Stationary, 


(b) First Order Markoy Process 
1X) has the structure 
Xi + n, е, 


lal<1, (2.3) 
where {e;} is purely random process, sa 


У, With m —0, 5 — 1. Further 
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e is not involved in X;, X, ,. ..., i.e. X, depends on ei, ез, ... e, but 
is independent of е; +, es .... 
Multiplying both sides of (2.3) by X, , and taking expectation, we get 
Су-е C40 
or, ра-- 4%, =0, be. 9, = — оу =a (say), |a| < 1. 
Proceeding in the same way, we get 


Pa} «уру =0, Le ро = (— %)? = a? 
Pa 2 pna = 0, 1.6: Ра = (— a)" = а" 


The process is covariance stationary. The representation (2.3) is a 
linear stochastic difference equation; Y, can be expressed as a formal 
solution of the difference equation as follows. 

Denote the backward shift operator by B, i.e. BX; = Xy. 
Then ВХ, = Ху and ВХ, = Xo k= 1,2, 3,.... 
Thus: the difference equation can be written as 
(1 + œB) Xi = e 

X, = (1 +8) е = 2 AS o3)* BY} e, 

(2.4) 


co 
= EX pk erk 
k=0 


(c) Moving Average (MA) Process 
Here (Y; can be represented as 
Х, == аре + A1 ё-1 +... + 4h Era, (2.5) 


where the a’s are real constants and (ej) is a purely random process with 


mean 0 and variance o?. X, can also be represented as 


h 
X,—f(B)e, where f(B)— Zeb; 


when ar 0, X, t & T is called an MA process of order h. 


We have Е{Х,} = 0. 
апа Cy = E{X. Xen} 
— Е{айкет + dida yt... + one Ds EIER 


+ Х aae) 
rt 


+ аһ-кар) o, К< 


= (аак -+ lkt shee 
koh. 


=0 


A necessary but not sufficient. 
representation of order / is poss! 


15(45-95/1982) 


condition that such a moving average 
ble is С, = 0 for k > h. We have 
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Ck "ay аак 1.2. d аһ-каһ . k 
ке айар. tap d^» 
=0, if h< k. (2.6) 


The process is covariance stationary. 
If, in particular, ак = 1/(h -- 1), k = 0, 1, .. < h, then 


hs - ERE 
Rasy =l hey OSk<h pn 
=0, otherwise. 


The correlogram consists of a linearly decreasing set of points. 

The representation (2.5) of X, consists of a linear expression with a 
finite number of terms in ё, ё»... rm It is said to be of order A. 
The first order Markov process (2.4) can be considered as a moving 
average of infinite order with p; = alkl for all к. 

Wold (1954) has given a necessary and sufficient condition that there 
exists ап MA model of order A Corresponding to a specific set of 
autocorrelation coefficients. Since Wold’s work, interest in this topic 
increased and several contributions have been made. Recently Davies 


et al. (1974) found general inequalities involving autocorrelations of an 
MA process. : 


(d) Autoregressive Process (AR Process) 
The process {Л} given by 
Xe bb ba iso X a en by 0, Q.8) 
where (e;) is a purely random process, with mean 0, 
regressive process of order h. 
X, can be obtained as a soluti 
equation 


is called an auto- 


on of the linear Stochastic difference 


8(B) X, =e, (2.9) 


h 
where 2(B)— X b,Br, b, 1. 
r-0 


Suppose that 9(B) = TI (1 — АВ), zz, de zl. 


distinct Toots of the equation 8(2) = 0. Further Suppose that |z;|< 1 
for all i, ie. all the Toots of g(z)=0 lie Outside the unit circle, 


i.e. the roots z; of the characteristic equation fa Ў b,zk-r = 0 (where 


f(z) = z-^g(z-1)) all lie within the unit circle, 
(2.9) can be written as 


‚лу. are the 


r=0 
The complete solution of 
h 
X, —534 z, er 
pga ШЗ er, 
where A, are constants, Now 


1 h 
zB) e; = u (1 — zB) e, 
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—2b,Be-—Zb,e nbl, 
r=0 r=0 
where b,’ are constants involving z;’s. 


If the process is considered as begun longtime ago, then the contri- 


h 
bution of z A.Z damps out of existence. X, is then given by 
X, = и ег: (2.10) 


Thus an AR process сап be represented by an MA, process of infinite 


order. 
The coefficients b,’ of e, іп the right hand side of (2.10) can also be 


obtained as follows. Using the expressions for X, as given .in (2.10), 
for 7 — 1, ..., t — h and then substituting them in (2.8), we get 
Eb/e, , + b, Eb, e-r + By È breiar + + + d br Бен = er 
Equating the coefficients of e; ei» .. - from both sides, we get. 
DESI ] 

b, + bibs = 0 

MA Bn ES mf 2.1) 
baat bibra t -e Бо 0 ) 
and bld bibir +... + brbr-a=0, r= h, h+1,.... (2.12) 


In other words, b; satisfies the difference equation (2.12) together with 
the initial conditions (2.11). Thus, b, сап be obtained by solving (2.1-) 
with the help of (2.11). 

We may put the result in the form of a theorem as follows: 


Theorem 8.1. If the roots of the equation 
Kz zz + Бг! +... +b, = 0 
all lie within the unit circle, then the autoregressive process 
X; bX +... + Xn D 
can be represented as an infinite moving average 
X= X b, ers 
r=0 
where b, are the roots of the difference equation 
оаа ose 
subject to the initial conditions 
pro 1, bf + bby = 0, by Hyb b+ babo = 0, -> 
Big Ба. + brabo = 0. 


Let us now determine the autocorrelation function of X;. We have 
ЕХ} =0. Multiplying (2.8) by Х- and taking expectation, we find 


that the autocovariances satisfy the relation 
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Ck + БС +... F by Cyn = 0, k = 1,2, .... 


Noting that ек = Cy Cs, 
we get 

Фк t Diosa +... + bnper = 0, k —1,2,... (2.13) 
or &(B) ex = 0, 


which is a difference equation in o, of order Л. These are known. as 
Yule- Walker. Equations. 

Now o, = 1, and putting k = 1, 2, -+-,4—1 in (2. 13), we get (a — 1) 
equations іп рү, ..., oj 4 Which can be solved in terms of the coefficients 
b,..., by. These will constitute Л initial conditions which could be 
used for the solution of the difference equation 


ек + Вур +... + biper = 0, k > h. 
The difference equation in ек (К >h 


Thus if the characteristic equation f(z) 
then 


) can then be completely solved. 
= 0, have distinct roots Zion. Ee 
h 
РЕ= Ў az (2.14) 

r=1 
where ә, are constants which сап be 
tions. The process is evidently covariance stationary. 
Note: The process (X) given by (2.8) cannot be represented by (2.10) 
when the roots of the characteristic equation Ҳг) = 0 do not all lie within 
the unit circle. In this case E(X) =~ A,z,! which is a function of t 


does not damp out even for large r and the process is non-stationary. 
Let us consider the case h — 2. 


determined from the initial condi- 


(e) Autoregressive Process of Order Two (Yule Process) 
This is given by 
Xi + BX + ЫХ, = er. (2.15) 


Assume that the roots 21, 25 of the characteristic equation 


2+ bz b= 
are distinct, and that both the Toots lie within the uni 
and 4b, > 6,2, Neglecting the contribution X Az for large t, we get 
X -[0—28) (1 — zB) e, 7 


= [1 + @ +4) B+ (t+ zz, лау вз...) 


t circle, i.e, b, < 1, 


е; 
E 
E (zH — zt) 
epe Ble, 
(2 — 22) 


=S bhp (2.16) 
r=0 


where b; = (g — 73 (2, — z,), when z, + Za 


= (r + 1)z,', when 2, = Z3. (2.17) 
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We get E(X;) = 0 and 
var (X;) = Z (b. var (е) 


___var (е) | Е 2 1 
(= 22)" [1 22 1 zn =| 
var (Ху) 1+6 
so that - = ree 
Е var(e) (1 20 (1 + b — 6)" 


The Yule-Walker equations are ex + Dy ека + Брка = 0, k = 1,2 
270, Poe aris 


pk = ^+. nez. 
++ Бър; = 0, 


whence we get 


Fork = 1, 
whence BEA "UE ЧЕ “furtlien py = 

{ DIA 14445 er ро = 1. 
we can determine the two constants оу and &,. 
(real or complex) of the characteristic equation, 


Using these two values 
Thus for unequal roots 
i.e. for 21 5 22 
zk+ (1 — 23) = zkt (1 — 218) 
о lk oca ee 
When the roots 21, 22 81€ complex, we can write z, = pe!*, 2, = ре-!, 
0« 0 < 2n, 50 that p — V (2122) = a/b, 2p cos 0 = 24 +- Zg = — b, 
or cos 0 = — b,J24/ bs. 
Putting these expressions for 21, z, in (2. 18), we get 
pe (gie (1 — pet) g ie) (1 = petit) 
e pipe»? 


p [2i sin (k + 1) 6 — 2ip? sin (k — 1)0 
= еер" 2i sin 0 


— pk (sin КӨ cot ф + cos k0) 


sin (k0 +9) 
Lime (2.19) 


where tanj— iH tan 0. (2.20) 
d real) root of the characteristic equation, ie.z,— 23 2, 
k(1— 2/1 + 2*)} zk, k=0, 1, 2, ..- 

ance stationary. As p < 1, ек — 0 for 
llatory and decaying in character. 


For equal (ап 
p= 01+ 


s clearly соуаг! 


This process i : à 
m is osci 


large k, The correlogra 
Moving Average Process (ARMA process) 


(f) Autoregressive х 
Consider а process paving the representation 
& кх = È ше Cae 

s=0 


r=0 


(2.21) 
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where (ej is a purely random process with mean 0, and the roots of 


I= $ b,z?- — 0 all lie within the unit circle. Such a process is 


called fan autoregressive moving average process and is denoted by 
ARMA (р, 4). We сап write (2.21) as 2(В)Х, = h(B) е, Where g(B) 
and A(B) are polynomials in В of degrees р and q respectively. Then Y, 


can be represented as an infinite moving average X, = Z v,e, , as follows: 


q r А 
Put u, = È а,е,_; and let the roots z,i=1,2,...,p of the equation 
s-0 


J) = 0 be distinct. Then X, = Ў Arzt 4- П (1 — 2; B) tu.. 


r=1 i 
Now as | z; | < 1, we have for large / (neglecting the terms X A,z;'), 


r=0 


Е co q 
a TT (1—2B)!u— X b'u,- X b,'( > аа) 
i=l r=0 


s=0 
= Y уе, (say). (2.22) 
r=0 


The v/s, r= 0, 1,2, ... in (2.22) can be determined by substituting the 
above expression for X, in the l.h.s. of (2.21) and then comparing the 
coefficients of e, , from both sides. 


We have E(X;) = 0. Multiplying (2.21) X, , (k > 0) and taking expec- 
tations, we get 


(В), = 5 brek = 0, k >a 
r=0 


Р 
whence Рк= È az k > Ф (2.23) 
г=1 


«, being constants. 
8.3 Time and Frequency Domain: Power Spectrum 


Covariance and correlation functions of stationary processes are functions 


in the time domain. It is found’ that the study of processes in the 
domain’ 


for the st 
This leads 
Stationar 
Correlati 


*time 
is neither simple nor illuminating. An alternative approach 
udy of stationary processes is through ‘frequency domain’. 
to simpler and more rewarding results. Consider a covariance 
Y Process in discrete time having Ck and рк as covariance and 
on functions Tespectively. 


poco 9f Covariance and Correlation Functions b 
9 e covariance (correlation) function is even, and positive semi- 
efinite, 


(2 The Correlation function has the representation 
т 


фк = | elt ДОУ (3.1) 


where Fi) asthe chase V 
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When ~F; is absolutely continuous. (and dF,(w) —fi(w) dw), one gets the 
inverse representation 


fw) = 


The corresponding representation of covariance function is 


l оо 
> рее, (3.2) 


2x k=- 


т 


Ck = | ei*» dF(w) (3.3) 
where dF(w) = CodFi0v)s when dF(w) = fw) dw, 
1 со 
м) == 7 Cp e”. 
K 2x е3 ке (3.4) 


The function Fi(w) (so also F(w)) is called the integrated spectrum; the 
function f(w) is known as the spectral density function and fi(w) as the 


normalised spectral density function of the process. 
The information contained in the sequence (Cj) is thus contained in 


the function fA) and vice-versa. From covariances which are ‘time 
domain' entities one gets a unique *frequency domain' function. 

The relation (3.1) is of the same form as that between characteristic 
function and distribution function of a r.v. As a consequence of non- 
negative definiteness of рь, the relation (3.1) follows from a celebrated 


theorem due to Bochner-Khinchine. 


Example 3(a)- White noise process {Xj}. We have 


E(X) = 0, 
Е (X.X3 = 052, г= 5 
= 0, rs. 


Thus C.=0,k #0, С, = о. We have 
6 2 
fiw) == КЕИ Cy ete» = s —n<we<n. 


Again, given fiw), we at once get С, from the inverse relationship (3 3). 


Autocovariance and Autocorrelation Generating Functions 
The autocovariance generating function (АСЕ) of a stationary process 


{Xj} is the power series 


сд = E Ce 2s 


r values of z for which C(z) is convergent). 


(considered fo 
Cy Co), one gets the autocorrelation generating 


Replacing Cx by Pk (= 


function 
5(2) = yey рк zt = С(@/б = С(@)/(уат Хо). 
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The spectral density f(w) can be expressed in terms of AGF as follows: 


fo) = x A. Cp e-ik» = = {Cle} 


Theorem 8.2. Let Y,= is aX; 


where У | а; | <œ, be a stationary process. If Cy(z), Cx(z) are the 
АСЕ? and fy(w), fx(w) are the spectral density functions of (Y) and (Xj) 
respectivelv, then 


Cy(z) = A(z) A(z) Cx(z) 


and ЖУ) = Ме!) h) Faw), 
where h(z) = РІ ajzi. 


Proof: Assume that E (X) = 0, then E{¥;}=0. Denote E {XXn} 
by C, and Е {Ү, Үк} by Ry. We have 
Re = Е (ҮҮ) = EQ EX) ( 2 a, X) 


со 


cs 
= Z У aja; Cu; 


j=0 Imo 
and hence 
Cy(z) = М > Ryz* 
SB AS Sua Gens. 
d NES 12 42 2, 4 Сы. ni 
=.5 5 5 J+ +1 
J=0 1=0 k=-00 АЕ Ces zh 
=( 2 а2)( = a; z) ( 5 C,z5), 
1=0 j=0 s=- 
(setting = фуу 
= h(z) ще) Cx(2), ар. 
Арап 


ЛО) = = (Cy (e) 


= 2 (ei) е») Cx(e-i»y 


= h(el") h(e-iv) tx). А 


Note: Assume (X) to be a white noise process; then th 
theorem can be applied to obtain th € above 


* spectral density function 
АК and ARMA processes. у Р 
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Example 3(b). МА process of order й: X, = i ajej, (e) being a 
white noise process having variance о?. We get jd 


ло) = (e) Me™) 


= (o"/2n) {шу a, €" d ah ell} (a, + a, ет!" 
+.. F ag е1). 


On simplification, we get 
h h—k 
fw) = (0/27) t E: d, Gran} COS kw 


h 
= (0/27) È Ске, 
k=-h 


where C, = E (X X „h as is to be expected. 
sive process Or Markov process: 


Example 3(c). First order autoregres 
hite noise process having variance 


X, = аха + ё la| <1, {е} being a W 
c?. The process Сап be represented as (see (2.4)) 


E 
X,= X dej 
j-0 


We get fxiv) = (z) {h (е) h (e), 


where һ(е'*) = ше" = 1/(1 — ае”). 
в? 


2т(1 — 2a cos w + а?) " 


І г 
Thus ЛО) = (i ae" —4е") 


order h: Xi + 4XCaG ss + OX = е, 
ess having variance o* and the roots of 
2.4 bi =0 all lie within the unit 


AR process of 
hite noise proc 
CROIRE 
e theorem to 


Example 3 (d). 
where {2} is à W. 
the equation (2) 
circle. Applying th 


h 
PLA bjXi-j, 
j-0 


дә = (b tems FO 


we get 
pectral density of (e). But fw) 


where f(w) is the 8 
at once get 


= 02/27, so that we 


flv) = o К H Ье») i bye?) 
2m j=0 / k=0 й 
h=2. Тһе spectral density function fx(w) of a 


Particular case: 


second order autoregressive process 


X, + bX- + b, s = ё 
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is given by А 
Лб») = E М + bye" + Беч") + bye + beej 


в? 1 
ре (=) 1 + by? + b” + 26,(1 + ba) cos w + 2b, cos 2w 
Example 3(e). ARMA (p, 4) process: 
Ў bX, = $ du Des]. 
0 r=0 


r= 


where {е,}:іѕ a white noise process and the roots of the equation 
f(z) = z^ + bz +... +b, =0 alll lie within the unit circle. Putting 


a 1 
Z,= È ае, 
r=0 
and applying the theorem, we get 
cs Sow Sb y 
fw) = ax (2 ае )( Eae Ni 
where fz(w) is the spectral density of the process {Z}. 


Again from 
= Ў Ь.Х,-,, we get 
r=0 


Fav) =(E bey È be-i) уугу), 
r=0 s=0 
Thus equating the two expressions for Jz(w), we get ' 
ка 2 a,eirv) . ( $ а,е-ізю) 
лор a ee on 
C3 Бет) $ һеч) 
r=0 s=0 


Continuous Parameter Processes 
Stationarity for a Process in continuous time is defined in a similar 
manner. A process {X(t), — 0 <1 < ©} ih continuous time is wide 
sense stationary iff, for all t, the mean value function E{X(t)} = m is in- 
dependent of t and the covariance function E [GX(t) — m) (X(t + y) — m} 
= R(v) is a function of the lag difference (t+v)—t=y, The continuous 
time analogue of (3.3) is given below. 
R(v) is positive semidefinite and has the representation 
Rv) = J e" dF(w), (3.5) 
" -e0 

where F is nondecreasing and bounded; when dF(w) = fw) dw, 


Ҳю) = = [ et R(y) dy. (3.6) 


=% 
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[The result follows from Bochner’s theorem; for its proof see Goldberg 
(1961)]. 


Example 3(f). Fora process having R(v) = ae-^"l, the spectral density 
function is 


1 f —iwv 
fo) = 55 | еті" ael dy 


dH | J gói dy + | еы)» av] 
2x 
0 


endis cdita, 4 1 
"2x M — iw ' b+ iw 
on ab 
BETIS) 


8.4 Statistical Analysis of Time Series: Some Observations 


Time series is one of the most widely pursued areas of statistics. 
Problems of statistical inference in time series, though very important 
from the point of view of practitioners, are beyond the scope of this 
book. However, we propose to give here very briefly an idea of the 
type of problems that arise. Г і 

The special feature is that the data are arranged with respect to time 
uccessive observations are considered to be not independent. The 

inference of such types of stochastic processes arising in 
ds does not differ in principle from that of other types of 
t the existence of some dependence in successive 
observations often implies that the classical methods become inadequate 
dequate extension. For example, the asymptotic theory of 
and need a ТЕ od estimates does not apply in general to dependent 
maxirum likel IT independent repetitions of the process are not 
observations. 4 single observed sequence is available and this also 
available; inde A s necessary. Difficulty arises because of the fact 
так a has so many more possibilities a priori than indepen- 
Meu ften than not, the distribution and the dependence of 
dence, and more 0 resenting stochastic processes may not be clearly 
the observations Ser theory of time series, like that of stochastic 
known. The А is stillin а comparatively early stage of develop- 
Eco к н: the results concern large samples, and small sample 
ment, and mo Е 
methods are less often available. in 

The inference problems cen one For 

with the parameters of the moce™- 


and s 
problem of 
diverse fiel 
statistical data; bu 


the parametric models are concerned 
example, for the AR model 


ib "LIS b —1, 


r=0 
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the problems include estimation of the b’s, the variance of е,, the distri- 

bution of the estimates and hypotheses testing and similarly for the MA, 

ARMA models. There has been considerable amount of work on 
estimation and the question is still engaging the attention of researchers 

(see Chatfield (1977), for a review of recent developments and for an 

exhaustive list of references). 

Nonstationary models have also been extensively studied (e,g. Priestley, 
Granger et al, Koopmans et al; (also see Box and Jenkins (1976) for 
building of stochastic models, specially non-stationary and their use in 
important areas of application). 

In addition to making inferences about the statistical structure of the 
underlying process, theanalysis of time series has two other purposes: 
forecasting and control. The Box-Jenkins forecasting method, based on 
a class of models called integrated autoregressive moving average 
(ARIMA) processes, has roused keen interest. They start with a modified 
type of ARMA model, such as g(B) X, = h(B) e, where some roots'(say, 
d2)ofg(B)are equal to unity and the others lie outside the unit 
circle. The corresponding nonstationary process (called ARIMA 
process) has the representation 


ДВ) (1 — By! X, = h(B)e;, 


where all the roots of fB) lie outside the unit circle. Introducing 
Seasonality components, Box and Jenkins (pp. 303-305) arrive at the 
general multiplicative seasonal model 


JB) F(B*) (1 — B (1 — BX, = hq B) Ho(B*) e, 


where (ej) is a white noise process, f», hg are polynomials in degree p, q 
in the lag operator B, Fp, Но are polynomials in B5, s being ‘the number 
of seasons per year. It is assumed that Jp» tq have all zeros outside the 
unit circle, A has no zeros on the unit circle and all the polynomials fp, 
hq, Fp, Но have unity as the constant terms. 
wide range of models, both Stationary and non-stationary. Box and 
Jenkins claim that this class is sufficiently flexible to provide a close 
approximation to the true model for many sets of homogeneous data. 
The main problem is the selection of an appropriate model. Box- 
Jenkins provide detailed techniques for identification of a proper model, 
for estimation of its parameters and for checking the adequacy of the 


model. Their techniques have aroused keen interest resulting in further 
active work. 


This class provides a 


Apart from the estimation and associated problems of a parametric 
model there are some other statistica] problems associated with a 
Stationary time series with mean р. Given a consecutive sample of 
observations X,, ..., Xr, one is concerned with the estimation of the 
mean, of autocovariances or autocorrelations, of the Spectral density 
and also with the large sample behaviour of the estimates (see 
Anderson (1971), Fuller (1976) for а detailed account). The problem 
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of estimation of the spectral density function has long attracted the 
attention of time series researchers (Bartlett, Blackman and Tukey 
Parzen, Grenander, Medhi, Priestley, Rao etc., to name a few; a 
Jenkins and Watts (1968), Hannan (1970) for some details). "тһе 
difficulties of selecting а best estimator оп theoretical grounds are still 
being appreciated (Bloomfield, 1976). 


EXERCISES 


84 Do the autocorrelation functions uniquely determine a moving 
average process? Discuss with the help of some suitable examples. 


8.0 Verify that the four moving average processes 
X,— 1.6 e, —0.8e4 + 0.4 e — 0.2 er-s 
X= — 0.2 е + 0.46, — 0.8 ei + 1.6 es 
X, = 0.46 — 0.2 er 4- 1.66, — 0 8 es 
Хх, = —0.8e + 1.66.4 — 0.2 egt 0.4 e з 


autocorrelation coefficients — 42/85, 4/17 
(Wold, 1954). 


hich have for autocorrela- 


all have the first three 
— 8/85 and the rest zero 

8.3 Find two moving average processes W 
tion pọ = 1, e = .4 ex = 0, k»2. 

8.4 A set of autocorrelation functions ро = 1, ра, <- Ph Of an auto- 
regressive process are given. Will these uniquely determine the 
autoregressive process? 

It is given that an autoregressive proces: 
correlation coefficients ро = 0, ру = % Pa 
regressive process. 

8.5 Consider the ARMA (1; 1) model 

X,—a, X4 7 6 b oy $= OE 1,42, ..- 
and Е(е,е,) = 0, r z^ s. 
n infinite moving 


s of order two has auto- 
= b. Determine the auto- 


where |a| < 1, Bed = E(e?) = т, 
Show that the model can be represented as а 
average 
Х,=е+( + а) UR at-! erk- 
process of order h given by 
. d- dg ењ a, 9, 
ots of the charac- 


8.6 Let X bean MA 
Yy E 110061110 
where (e) is 2 purely random process. If the то 


teristic equation 
Esta суса ЧЫ: + a,=0 
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8.9 


STOCHASTIC PROCESSES 


all lie within the unit circle, then X, can be expressed as an auto- 
regressive process of infinite order 


о 
> сХ, =en 
г=0 


where the coefficients с, satisfy the difference equation 


Cr 4- Cra + --- + аһ Сов = 0,r=h,h+1...., 


and the initial conditions 


Co = 1, c + a, =0, с + 4,0 + a,=0,..., 
i, Choa + dy Cho +... + Op =0 (Fuller, 1976). 
Let Х, be an ARMA (р, q) process given by 
g(B)X, =f(B) е, 


where {e,} is a purely random process, 
(В) = 5 bb, by =1, 
r=0 
q 
S(B) = È a-B', а = 1 
r=0 


and the roots of g(1/z) — 0 all lie outside the unit Circle. Show 
that Y, can then be represented as an MA Process of infinite order 
X,— X се, : 
r=0 
Consider the process 
Xi— ах +b 4- e; 


where {е} isa purely random process with variance unity. Further 
assume that the process was started at time 0 and that Y, — 0. 
Show that i 


and C, — “( i =) 
Deduce that, if the series was Started long ago, and if [а|<1 
Ск = a*[(1 — a). 

X a,X,-, + AX ta =й; 

where Y, b,Y, 4 +5, Yis 
and {е} is a purely random process and | 

FEL Guia оа et the roots of m? + ат 


Р 2=0 be less than unity ; 
value. Obtain the spectral density function of d ipi ups 
Jg. 


Let 


—en 
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8.10 Let Z: = X, + Yn where (X;) and {У} are tw i 
e ( o stationary proce: 
with spectral density fx(w) and fr(w) respectively. Then son 
that {2} is stationary and has spectral density fz(w) = fx(w) + 
fr(w) Hence find the spectral density of 


Zi =X, + ey 
where X, -paX,-,— w, and {е} and {v} are two independent 
white noise processes. 
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CHAPTER 9 


BRANCHING PROCESSES 


9.1 Introduction 


The history of the study of branching processes dates back to 1874 when 
a mathematical model was formulated by Galton and Watson for the 
problem of ‘extinction of families’. The model did not attract much 
attention for a long time; the situation gradually changed and during the 
last 40 years much attention has been devoted to it. This is because of 
the development of interest in the applications of probability theory, in 
general, and also because of the possibility of using the models in a variety 
of biological, physical, and other problems where one is concerned with 
objects (or individuals) that can generate objects of similar kind; such 
objects may be biological entities, such as human beings, animals, genes, 
bacteria and so on, which reproduce similar objects by biological methods 
or may be physical particles such as neutrons which yield new neutrons 
under a nuclear chain reaction or in the process of nuclear fission. 

We consider first the’ discrete time case. Suppose that we start with an 
initial set of objects (or individuals) which form the 0-th generation— 
these objects are called ancestors. The offsprings reproduced or the 
objects generated by the objects of the O-th generation are the ‘direct 
descendants’ of the ancestors, and are said to form the Ist generation; 
the objects generated by these of the Ist generation (or the direct descen- 
dants of the Ist generation) form the 2nd generation, and so on, the 
direct descendants of the rth generation form the (r + 1)st generation. 
The number of objects of the rth generation (r = 0, 1, 2, ...) is a random 
variable. We assume that the objects reproduce independently of other 
objects, i.e. there is no interference. 

Definition. Let the random variables X,, Ху, Х,, ... denote the size 
of (orthe number of objects in) the Oth, Ist, 2nd, ..., generations 
respectively. Let the probability that an object (irrespective of the 
generation to which it belongs) generates К similar objects be denoted 
by рь where py > 0, k = 0,1, 2,...,®рк= 1. 


The sequence (X, n—0, 1, 2, ...} constitutes a Galton-Watson 
branching process (or simply a G.W. branching process) with offspring 


distribution ( px}- ^ 
Our interest lies mainly in the probability distribution of X;. and the 


probability that X, ->0 for some, ie. the probability of ultimate 
extinction of the family. 


46 (45=95/1982) 
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Note 1: Unless otherwise stated, we shall assume that X, = 1, i.e. the 
process starts with a single ancestor. 

Note2: The sequence {X,} forms a Markov. chain with transition 
probabilities v ; 

ру = Pr(Xs4 —Jj| Xa =i} 5/—0,1,2, ....* 

Jt is however not always easy to specify pij. 

Note 3: The generating functions prove very useful in the study of 
branching processes. 


9.2 Properties of Generating Functions of Branching Processes 
We have 
Xn 
Xan YU, Q.1) 
where C, are i.i.d. random variables with distribution (pj). Let f 
P(s) = z РЕ, = kst У pest (2.2) 
be the p.g.f. of {¢,} and let 
P,(s) = ZPr(X, LASS 0-—0,172; Oh 
be the p.g.f. of {Xn}. 


We assume that X, = 1; clearly P(s) = s and P,(s) = P(s). The r.v.’s 
X, and $, both give offspring distribution. 


(2.3) 


Theorem 9.1. We have 
P,(S) = Р, (Р(з)) 


(2.4) 
and P,(s) = P(Py_\(s)). (2.5) 
Proof: We have 


Pr{X, = k} m Pr(X, = k| Xn =j} Pr {Xn =} 
со 1 
=2 Pr{ X C = k- Pr(X,,—j) 
=0 r= 
so that, for n =1,2,... 


P,(s) = X Рг{Х„ = k)s* 
k=0 
сө со ) 
к 2 d 2, Ent EA Gr = K) Pr {Xn = J 


= BP MER ++... киеу 


The expression within square brackets being the 
Уч p.g.f. of the sum 
014... + 0 ofj iid. random variables each with p.g.f. 
[ECP Thus PT PG), equis 


Pafs) = E Pr {Xna = j) [POY 
= Р„(Р(5ў). 
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Thus we get (2.4). Putting n = 2,3,4,..., we get 
P2(s) = P,(P(s)) = P(P(s)), Ps) = Р.(Р(5)), Pa(s) = P,(P(s)) 
and soon. Iterating (2.4) we get 
P,(S) = Р»_, (P(S)) = Pn- (P(P(s))) 
= Pn-z(P,(s)). (2.6) 


For n = 3, P3(s) = P,(P,(s)) = P(P,(s)). 
Again iterating (2.6), we get 


Pr(S) = Р,_,(Р(Р,(5)) = Pn-a(P,(s)), 
and forn=4, P,(s) = P,(P,(s)) = P(P,(s)). 
Thus Р(5) = Paz (Px(s)), k = 0,1, 2, ..., п 
and fork =n—1, 

Pr(s) = Py(Pn-(S)) = P(Pna(s)). 


Thus we get (2.5). A 
Note that even when X, = iz 1, the relation (2.5) holds but (2.4) 


does not hold. 
Moments of X, 


Theorem 9.1 could profitably be used to find the moments of X,. We 
have Р'(1) = E(C,) = E(X3) = m (say). 


Theorem 9.2. If m= E(X,)= X kpr, and c? = var (X3) then 
k=0 


E(X,) =m" (2.7) 
л—1 n 
and var (X,) — тт) а if mal (2.8) 
= по? if m=1. 


Proof: Differentiating (2.4), we get 
Py, (s) = Psi (P(s)) P'(s) 
whence Pa (1) = Рл-(1)Р'(1) = mPa- (1) 
and on iterating i 
P, (1) = m? Pas (1) 
= m1P'(1) =m". 
Thus E(X,) = Pa) =m" 
d proceeding in a similar fashion, one 


ting (2.5) twice an 
"n. Mir deni moment P,'(1), and thus the variance of X, in the 


2.8). ! 
cus Bi MS proceed to get higher moments of Xy. 
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Alternatively, the mean and the variance of X, can be obtained by 
noting that X,4, is the sum of a random number of i.i.d. random 
variables, and using standard formulas. 

Letm#1. We сап use the Corollary to Theorem 1.3 to find E(Xn+1). 


Since Xna = Fo we have 
Е(Хьы) = E(G) E(X») = mE(Xn). 
The solution of the difference equation is given by 
E(Xn) = Cm", n 21,2, 3, .... 
Since E(X)) = Е) =m, C —1. Thus E(X,) = т". 
Using the result given in Exercise 1 .6, we get 
var (Х,а) = E(X») var (C) + LECT" var (Xn) (2.9) 
= m'a? + т? var (Xn) (2.9a) 


We can find var (X,) from (2.9a) either by induction ог by solving 
the non-homogeneous difference equation. We employ the latter 
method. 

A particular solution of the difference equation (2.9a) is given by 

с?т" 
var (Xn) = о 
and a general solution of the homogeneous equation corresponding to 
(2.9a) is given by var (Xn) = A(m*)", where A is a constant. Thus the 
complete solution. of (2.9a) is 


var (Xn) = A(mby + € (2.10) 


_ Noting that var (X3) = 0°, we get A = о2/[т (m — 1)] so that (2.10) yields 


es vulc 
e noa... 


The result holds for all n and m Æ 1. 
gets the corresponding result for m — 1. 


When m='l, var (X,) = пв%. 


var (Xp) = 


By taking limit as m — 1, one 


When m=1, the variance of X, increases linearly and when m > 1 
(m < 1) it increases (decreases) geometrically with л. 

According as m<1,=1, or > 1, the Galton-Watson process. is 
referred to as subcritical, critical or supercritical respectively. 

We now come to the problem originally posed by Galton. 


,9.3 Probability of Extinction 


Definition. By extinction of the process it is'meant that the random 
sequence {Xp} consists of zeros for all except a finite number of values 
ofn. In other words, extinction occurs when Pr (X,— 0), for some 
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value of s. Clearly, if X, = 0 for n -- m, then X, = 0 for n > т; also 
PrXss e OP д = 0} = 1. ^ 


Theorem 9.3. If m < 1, the probability of ultimate та is) 15 afi 
m > l, the probability of ultimate extinction is the positive root less 
than unity of the equation > 
KE Р). | eD 
Proof: Let qa = Pr {Xn = 0), їе: 9, is the, probability that extinction 
occurs at or before the nth generation. Clearly 4, = P,(0), qı = Р,(0) = 
P(0) = p, and from (2.5) ; | 
| Яп = P(qni): ~~] (3.2) 
If p, = 0, then q, = 0, q =0, ..., ie. if the probability of no offspring 
is zero, extinction can never occur. If p, = 1, then q, =11, 4 = 1, ..., 
i.e. if the probability of no offspring is one then the extinction is certain 
to occur right after the Oth generation. So we confine ourselves to the 
саѕе0 cp, «l. £^ _ E y 
As P(s) isa strictly increasing function of s, qa = P(q,) = Р( ра) > P(0) 
== ф. Assuming that gn > q;-,,We get Yni1 = P(gn) > P(Gn-,) = qn and 
by induction 9, < 4, < 43... The monotone increasing sequence {gn} is 
bounded above by 1. Hence qn must have a limit lim gn = q (say), 
0<4< 1; 9 іѕ ће probability of ultimate extinction.” From (3.2) it 
follows that q satisfies q = P(q), i.e. q is a root of the equation (3.1), 
NE = = Р(5). | 
We now investigate further about the root. First, we show;that q is the 
smallest positive root of (3.1). Let sy be an arbitrary positive root of 
(3.1). Then а= P(0) < P(s) = So and assuming qm < So we get 
len Plam) < P(S) = 8 and by induction qn < Sq for all n. Thus 
= lim gn 5p which implies that q is the smallest lpositive root 
n00 
of (3.2). | А E 
For this, we consider the graph of y = P(s) in0 < six 1; it starts 
with the point (0, ра) and ends with the point (1, 1); the curve lying 
entirely in the first quadrant, is convex as P(s) is an increasing function. 
So the curve у = P(s) can intersect the line y =s in at most two points, 
one of which is the end point (1, 1), i.e. the equation (3: 1)jhas at most 
two roots, one gf-whichis-unity.-Twocases—are-nowÀ to be consi- 


dered (see Figs. 9.1 and 9.2). 
Case I. The curve y= P(s) І 
case (1, 1) is the only point of interse 
of з= P(s) so that q7 lim gn =t- " 
| pd) А Los POE lord: 
- ug PO) 9 £9, 48 Puy {8,9 
` -lim coe КЕЗЕ Uu 


lies entirely above the line y — s; in which 

ction, ie. unity is the unique root 

Theni) vwo 9d T AL sD 
i oF [ON 6 15113 rou; 


so that 
su 899 i, : ОЕ 
Thus „p rui sondiin qi 22 when руле ихо ONS © йк: 


noo 
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(1,1) 


Fig. 9.1 (m< 1) 


Fig. 9.2 (m>1) 


Case 11. The curve у = P(s) intersects У = 5 at another point (8, P(8)) 
such that 8 = P(8), è < 1, i.e. there is another Toot of (3.1), 5 < js the 
curve y — P(s) being convex, lies below the line У = 5 іп (8 1), and 
above y — s in (0, 8), i.e. P(s) < sing «€ 5 < land Р(х) 5 in 0 T) 
< $. Then q, = P(0) < Р(8) = $ and assuming Gm 
P(qu) < Р(8) = 8 and by induction Ф < 8 for all n. 


so that q — à < 1. 


< 8, we get dingy = 
Hence lim д, = 8 
n-o 


BRANCHING PROCESSES 247 


Now by the mean value theorem considered in the interval [8, 1], there 


is a value č in 8 < Ё <1 such that P’ (Qi XD RD 1, and as the 


derivative is monotone P'(1) > 1. Thus we find that 4 is the root less 
than unity of (3.1) when m — P'(1) > 1; 
We have thus proved the theorem completely. A 

Note: Thatq is a root of s = P(s) can also be seen by conditioning 
on the number of offsprings of the first generation. We have 


q — Pr (ultimate extinction) 
= X Pr {ultimate extinction | Y, = k}. Pr (X, = k}. 
k=0 
Given that X, = k, the population will be extinct iff each of the k families 
started by members of the Ist-generation becomes extinct. It is assumed 
that families behave independently, hence 
Pr {ultimate extinction | X, = k} = q*. 


Thus q— X 4 py — Р(д). 


Theorem 9.4. Whatever be the value of E(X1) = т, we have, as n > o, 
lim Pr (X, = 0} = q and lim Pr (Y, = К} — 0, for any finite positive 
integral К. 

Proof: We first show that Jim P.(s) = q, from which the above 
result will follow. 
Consider.the case т < 1, when Р(х) = s has the unique root 4 = 1. In 
0<s<q, P(s)«P(q) ^q, and Р) < P.(q) = P(P(P)) = P(q) = q. 
Assuming that Р„(5) < q, we get Pm+(S) < g and by induction PS) < q 
for all. Again P,(s) > P,(0) = qn; thus Gn < Р(з) < q. 
Hence Jim P(s)-—4 0<5< 4. 


Consider the case т > 1, when q is the root less than 1 of Р(Х} x In 
q < 5 < 1, the curve y= P(x) then lies below the line y = x, and q < 
Р(з) < з < 1. Again Pa(s) = P(P,(s)) = P(P(s)) > P(q) = g. Assuming 
that Р„(5) > 9, We get Р,+1(5) > 9, so that, by induction Рл(5) > q for 
alln. Again P,(s) = P,(P(s)) < P,(s) and assuming that P,(s) < PAS), 
we get P,(s) < Р,_1(5) for all n. 

Thus in 4<5< 1, 

q < Pils) < Р, 1(5) <... 


so that lim P,(s) > q. 
noo 
Suppose, if possible, that lim p,(s) = «> 4, then P(x) < о, and 
noo А P n 
lim Р,.1(5) = lim P(P,(s)), and we get a contradiction which is due to 
пу оо п+оо 


Our supposition that « > 4. Thus 
lim P,(s) = 4. 
moo 
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Só;^whátever be- Tue value or. Ey 2mm lim Pe Sn NS SGT SUE 
of's (өг! all s <1). An other words, ‘for! 2 $ р 1 


E Б Pr (X, ly de q. 


n9 k=0 


This ; implies t that, the coefficients ‘or. sf for ji za quem deem for 
infinitely large K) all ‘tend, to 0, „ҮР the, constant , term tends to; ho 
Thus, as n— co 
Pr (X, = k} 0, for nde positive integral k, 
and 4} 19 = Pr {Xp 031-7. d 
Since P,(1) = 1, we have, asn > оо о, ; | vii 
rabies Jf . Janie саа Еі sty Toii «v (i8) 
Note: The above result also follows from the "general theory of 
Markov chains applied to the’ chain’ (X), for whicli each of the states 


К — 1, 2, 3. ... is transient while the state 0-15 absorbing. We shall, 
now Consider another interesting result. 


Tlieorem:9.5. Ме have, for r, ^ 50,1, 2, 7. 
pez p ы | ХЫ: AUS 
Proof... For = laand л =.0, 1, 2,4). зме have 


] mil 
G4), 


u iue 
SE nn | Xn} = = Ft I Se | Xn} = z Eté kn 


(9) = (SA) КЕТ 


Assunie "that a. 4) holds ' forr = 
the Markov nature of 25 We get 


E Usus a | P = ETE {Хан | Хв - Excuse) Xal 


ba х aff ТЕ кы |, ay] | XJ : p n 


\ экиппёг/ 
= hi then E IX ed | к = ey Noting 


үш (ъ)% « (6838 {ты | Xn} =н {Хш} 
19] < (2).А noitoubni Хут ( } 
so'that the result holds ТОГУЛ R24 Cr (ул), (уп! Todd 


\ 101 ( 


Thus by induction, we have the result, A 
Note: - Another variable of interest is И, == X,/m", n — 0, 1, 2, 
{W,} forms a Markov chain. We'have Е {Wy} = | and for m> 1, 


E (QW, LI gays Bis (me — 1) ot 
А (хуз Ж A ) sp У т 1) і 


Wise m 
Dividing both sides of (3.4) by т"+т, we pm : 


E {War | Wn} = Wa j (3.5) 


iin с?., | \ ш 
=з D fe ge ! eer е 
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and since {W} is also a, Markoy chain... sii br 
Ee | Wr as Lg nes Bw 


"ue have weld tiken. aii эпн(1 < 


| Example 3(a). «Let py, & =.0, 152 ‘be*the probability ‘that ап individual 
-sinvagenerationsgénerates ki offsprings.! üsThém Рф) = pp pi geqiop nee, 
PEG Pys)o£. 3(5) cambe cálculated by''simplé'‘algébra. "The  pióbability 

Icofnextinction.is;one ifi mus 1; if m 221; itis given’ by thé root зек nl 
of ѕ = P(s). Suppose p, = 2/3, p, = 1/6, p, = 1/6; then‘ wayyy, 
The equation s = P(s) becomes s im 55-54 = 0. with roots l and 4; the 
probability of extinction is 1. ' Suppose P= = : 1/4, Px = 1/4, p, = 1/2; 
then m = 5/4 > the, equation.» = P(S) haş the, roots; 1/2;;and: k:sdThe 
root 1/2 smaller than unity gives the probability of extinction. Note 
that the probability of extinction 18- рь/р, or(4),according as po <p, or 
Po > p, and also that po < (or 2) Р» ifm (о <)1. 


.» lo 2noijontil 316 46 («ү end ene 5190W 


; Example, 3(b)., Let the probability. distributionaof the; ШЙ of! off- 
;,Spripgs generated by.an individual їп a.generation-be Poisson iwithi:meàn 
^, ie. P(s) = = ећ№-10), [t can be easily... seen, that, thes graph-of- P(s) in 
0<5<1 (ie. between the points (0, e^) and (1, 1)) is convex, and that 
„ће curve of Du = Р(5) always lies a above Jm 1 when, a iS 1, there being no 
‘other root of S= = P(s) except unity i in (0, Dt ie prob oba у. 106 extinction 
is then 1. When à > 1, the curve y P(S) intersects у= in another 
point whose s-coordinate has a value < l'and the probability of extinction 
will be this value of s.:For example, if 2 3- 2, 'й Сап be seen that s = e%(s-)) 
has a root approximately equal 10.0. 2, which j ds smaller than. 1, апд, фе 
probability of extinction is q PSU lv 


5 (2) 

Example 3(c). Let the distribution of the number of offsprings be 

d m with py—b(1 — 5b) k=0, 1,2, ... (0b < I). »»u Then 
= (1 — b)/b and P(s))= b/(1 —sQ1— D). The. equation s= P(s) has 

the roots 1 and b/(1\— b. If m <1, then the probability of extinction 

is 1; if m > 1, the root b/(1 74), «and the;probability.i:of: extinction 

is equal to the root ba — b). 


Г 


i 
Example 3(d). Let рк = b т k=l} 2,..., 0< b, с, Б+с<1 and 
= sono 
p,—1— È рь Тһепт= (1-с), | We baye QA 
kel ees g 


bi Bs I —G)A к 
ROTI ee (0) 


The quadratic equation LO! has, he roots Ts 
4 1-029 


1 and ET ОЛЕ 3 E» py: 
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If m = 1, then s, = 1 and the`probability of extinction is 1; if m> 1, 
So < 1, and the probability of extinction is 4 = s,(< 1). 

This model was applied їп а series of interesting papers by Lotka to 
find out the probability of extinction for American male lines of descent. 
The values estimated by him (in 1939) from census figures of 1920 give 
b = 0.2126, с = 0.5893 (m ~ 1.25 > 1) and the probability of extinction 
q = S = 0.819. l А 

Note: Itis not always possible to put the generating functions P,(s) 
in closed form. The generating functions P(s) obtained in examples 
3(c) and 3(d) are of an interesting form; they may be considered as 


particular cases of the more general fractional linear form (or general 
bilinear form) 


oe eee 
P(s) “үр 9 — 81540. 


When P(s) is of the above form, P,{s) is also of the same form 


3 


_ Zn + Bus 
yag Fc 

where an, Bn, үл, 8, are functions of о, B, y, 8. 
Further, it may be noted that the equation s — 
fractional linear form) has two finite solutions 
So <, =, > or l according as т = Р), 


P(s) (where P(s) is of 
1 and s, and that 
=or< T. 


Example 3(e). P,(s) for Lo 


tka's model considered in Example 3(d) 
above. For any two points 


и, у, We get 


Р(5) — Р(и) 5—и1— су 
Р() — Р(у) 5 ус 


Put и = sẹ» v = 1, then Р(5,) = So, P(v) = 1, so-that 


P()—s _ s— 5, l—c 
POSI s—1I-cs 


1—с a [2-a P(s) —1 
l5 22—180 (5—1) 
Let т 5 1, then taking limits of the right hand si 


whence 


de as s — 1, we get 


1—c КеП 
leo ow 
Hence 
P)-—-s [lYs— So 
у= (a) Ft 
Thus 


P.(s) — s, — P(P(s)) — s, —(1)\ Pls) — зо 
P,(s)—1 P(P(s) — 1 (5) P(s) — 1 


=(1)\s—s5 
(Ow s—I 
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and on iteration 


P.(s)—so  (lYs—s, 
P,(s) = 1 = (5) ДЕЕ 


Solving for P,(s), we get 


деў 
Рд) = 1 — т (mas) + - (=): Salm 1, (3.8) 


23n6 -i- (1—e — nc) 
and P,(5) =т= (3.9) 


c--(1— be 


If т = 1, then s, = 1, and Р(5) =— = 


Limiting Results 
Suppose that m= 1, then в? = 2c/(1 — c), 


n Pr(X, > 0) —n(1— P,(0)} = TUS 
. —c 2 
and lim nPr(X, > 0} = Ка (see Theorem 9.8(а)). 


Suppose that m < 1, then so > 1 and 
lim m-"Pr(X, >0}=lim m-^(1— P,(0)} 
n-oo Doo 


17 = 
= lim —— о. NEUES 
pemn wise 


Again ЕРИК, =k X 2 0st 0 EDD 


1 
Thus lim рр, e|» stes (t -if-i) 
0. 


noo 


1 Vea 
and lim Pr, ok» 9-(1- s) 5-2. 
In other words, for large, the distribution of (X), given X,» 0 is 
geometric with mean so/(so — 1) and p.g.f. 
(1 = 1/s) s 1/59) s. 
bs) = пу 
ily verified that b(s) satisfies the equation 


It can be easi 
Ь(Р(5)) = mb(s)++1—m (see Theorem 9.9). A 
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9.4 Distribution of Total Number of Progeny 
Let X, denote the size of nth generation, n = 0, 1, 2, .. ; 9 # 
Then the random variable 
п 
Bf «® з= E Ke LEA ж. зк XG (4.1) 
dte 
5 denotes the’ total number of progeny, ie. the number of descendants 


upto and incliding the/nth generation and also including the ancestor. 


Є 
Theorem 9.6. The p.g.f. А.(5) of Yq satisfies the recurrence relation 


1 (s Кз) — 5P (Rn (5)), (4.2) 
(R.F = = “ЖӨ! . 
i P) being the p.gefiiof the offspring distribution. 
Proofs Теё Z,=X,4-.3. F Y, and G,(s) be its p.g.f. Then 


R,(s) = sG,(s). We have 
Ga(s) =)E Pr{Z, js 
k= ~ 


Now by conditioning on the size Xi of the Ist generation, we get 


Pr{Z, = k} = У Pr{total number of descendants in the succeeding 
i=0 


(n — 1) generations following the Ist is k — i| Xi 

If the process starts with one ancestor 
r descendants in succeeding m generations is the coefficient of s” in (5); 
and if it starts with i ancestors then the probability of having r descen- 
dants in the succeeding m generations will be the coefficient of 5" in 
[Gs(s)]. Thus 


= i Pr(X, = i). 
then the probability of having 


Pr(Z, — k) == [coefficient of 56-7 in (Gr. (5))] p; 


1 


ES [coefficient of s* in 6G, (S)y] 
= coefficient of sk in X Bi {50,1 (5)}! 
i=0 


= coefficient of 5* in P (sGn_, (s)). 
Thus G,(s) => Pr {Z, = k} sk = P (SG, ., (5)), 


whence R,(s) = SG,(s) = sP (К„_\(5)). 


Hence the theorem, A 


From the recurrence relation (4.2) it is theoretically possible to cal- 
culate R,(s), R,(s),.... We 


are however interested in the asymptotic 
behaviour of R,(s) for large n. ш 


Theorem 9.7. We have 
lim Rs == HY; 
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со 
where H(s) = eet isthe generating function of a sequence of non- 


negative numbers px. 
The function H(s) satisfies the relation 


H(s) = sP(H(s)), O<s<1; (4.3) 


further H(s) is the unique root of the equation 
pessT) (4.4) 
such that H(s) < E, where Ё is ће smallest positive root of x = P(x) and 
that H(1) = Z px = £. 
Proof: We have, for 0 < s.— 1, 
R2(s) —sP(Ry(s)) < sP(s) = R,(s) 


and assuming that Rm < К (5), we get 
Rma(s) = SP(Rn(s)) < sP(Rm—1(S)) = Rm(s) 


and hence by induction A,(s) < Rn-,(s) for all n >0. Thus fors < 1, 
{R,(s)} is a monotone decreasing sequence bounded below. Hence 
lim R,(s) = H(s) exists. From the continuity theorem of p.gf.’s, it 
no М A ix 

follows that H(s), being the limit of a sequence of p.g.f.’s, is the gene- 
rating function of a sequence of non-negative numbers рк such that 
HQ)-Xe 1. 

Taking limit of (4.2), we get 
T H(s) = sPH()), Om е <1, 


for some fixed s (in 0 < s < 1), H(s) is a root of the equation 
1 —sP(). 

fixed s < 1, y= sP(t) is a convex function of t and the graph of 
the line у = ѓ in at most two points. Let & be the 
of x = P(x); clearly č < 1. The function t — sP(t) 
0 and positive for t = Ё, and remains positive for 
and 1. Thus г = sP(t) has exactly one root between 
0 and & and no root between Ё and 1. The unique root of t = sP(t) 
equals H(s) and thus H(s) < &. Clearly H(1) is a root of х = P(x) and 
since & is the smallest root of this equation, H(1) = 5. Thus the theorem 
is completely established. A 

Note: Н(1) = 100г < 1 depending-on whether 14+%4+%+ .-- is 
finite or гої with probability one; and H(1) = 1 whenever m < É 


ie. 


For 
у =sP(t) intersects 


smallest! positive root 
is negative for t= 
values of t between č 


9,5 Conditional Limit Laws 


A. Critical Processes 
In Example 3(e) we o 

here some general results. : 

process. The probability of extinc 


btained some limiting results. We shall obtain 
Consider a critical (ie. with m = 1) G.W. 
tion is 1. Thus from Theorem 9.4, 
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we get Pr {Xn > 0} = 1. We also have var (X,) = no? > оо. The distri- 
bution of X, given that X, > 0 is of considerable interest. 


Lemma. For a G.W. ‘process with т = 1 and o? < со, we have 


NT 1 1 gi 
Lu 55] T (5.1) 
uniformly in 0 < s < 1. 


Proof: LetO0 < s < Тапа P'"(1) < со. Using Taylor's expansion 
of P(s) in the neighbourhood of І, we get 


Р(з) = + zd — s)? + r(s) (1 — s), ) 
where r(s) + 0 ass > 1, 
Thus ] | ROR я 


ГЕ Pe) Ts ~ (13) (Ps) —3) 


=i 5 {5 + п | 


Жү (5.3) 
where R(s) -> 0 as s — 1, and R is bounded. Again using (4:6), we get 


P,(s) = P(P(s)) = P(s) + T (1 — P(s))?+- r(P(S))(1 — P(s)* 
so that we 


1 1 c? 
E= P) грр) T + RPO) Cn 
and 
1 1 1 азау it 
2 {i =P (Se is | Eo % 2 Кн ROG). 


Iterating one gets 


1 1 1 mote, {т=з 
monne ty 2 ROKS). 


“A kag 
Since P,(0) < P,(s) < 1 and Pn(0) — 1 from the left 
of Pa(s) > 1 іѕ uniform. Hence the lemma. 
We shall now use the lemma to establish the following interesting 


limit laws (of which (a) ‘is due to Kolmogorov and (b), (c) are due to 
Yaglom). 


› the convergence 


Theorem 9.8. If m == 1, с? < со, 
then 


(a) lim n Pr (X, > 0} = 2, 
nc с 
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(Ы) іт ze x,>0}=$ 
n» n D 
(с) “im nU | Xn > 0b - e (— =) T 


Proof: (а) nPr(X, > 0)—n(1— P,(0)} 


-[{г=к®—'} 


Thus from the lemma (taking s = 0), we get 


3 Teo Or 2 
lim n Pe (a > 9) fim [77] =ч" 
(b) We have 


1 = ЕХ) = Е{Х, | as > 0}.Pr (X, > 0) + 0.Pr (X, = 0} 
so that 


Е{Х„ | Xn > 0) = FULL = , (from (a). 


Thus lim в | | Xa o} == 
п» ә 
(с) Let и> 0, and 
о[и < rn > Of (5.5) 


then taking L.T. (see equation (3. 1a) Ch. 1), we get 


Xe o}. (5.6) 


exp (— au) dF(u) = {exp (— Xe à) 


КЕСТЕ 


-H 1. Pr {Xn = 0) 


E(s*") is the p.g.f. of X,, we get 


E 


E 
| 


апа вїпсе P,(S) = 


ЕЕ зе PUT US 


ps Р ү ats) |x 4,0] - eo 


1 — P,(0) 


Zp "E 
"m: eS m, 6.7 


Nowas n> © 


n {1 — Pn (s»(-3)) c 2 (from (а)) 


and from the basic lemma (because of uniform convergence), we get 
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1 1 1 = І } 
п{1— P,(exp(— «[n)) zh — Pa(exp (— a/n)) 1 — exp(— z/n) 
1/п 22. 1 
i I —exp (— «п) ” 2j + a 
Thus from (5.6) and (5.7). we get, as n — со 


2 
Í exp (— au) dF(u) > 1 — am x 
LU 
vom 
11 2 00/2" 
z 2 20. 1 
5їпсе Т.Т. of > exp (- ES is Dam 


we have 
Т Xn 2 2и 
Ди рг {и < Ж <u>du| xol =: сею (– 20). 
which establishes the exponential limit Jaw. A 


B. Subcritical Processes 


Theorem 9.9. ( Yaglom's theorem). For a Galton-Watson process with 


т< 1, 
lim Реа) | X05, 212... (5.8) 
noo 
exists, and {bj} gives a probability distribution whose p.g.f. 
В(5) = EZ b;s! satisfies the equation 
j=1 
B(P(s)) = m B(s) -1 — m (5.9) 
or, 1 — B(P(s)) = m (1 — B(s)). (5.92) 
Further, EU = 1/9(0), where 9(0) = lim Pr Ds 0)/m^. 
= noo 
Proof: Using Taylor's expansion around s = l, we get 
к= 1— m1 — s) 4. (1— s)r(s),0<s<1 (5.10) 
or, TI = т — (у). 


(5.10a) 


Consider the function r(s)in0 <s <1; we have r(0) = m — (1 — Po) 20 


and lim r(s)=0. Further as P(s) is a convex function 
51-0 
, 1— P(s) 
Р 
(5) < SITES 


so that r(syss c sy [49 -— Pc) <0, 
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Thus r(s) is monotone decreasing, isbounded above by m and r(s) — 0 as 
$—1. Replacing s by Р,_1(5) in (5.10a), we get 


1—Pi{s) . 
TOP m (1 — r (Pi a(3)/m). (5.11) 
Putting K=1, 2, 3, ..., n and taking products of both sides, we get 
1 — P,(s) sci 
LOBO =m" HW {1-r(Pa(s))/m (5.12) 
; 1— P,(s)| . T. 
Since 0 < r/m < 1, the sequence md - s) is monotone decreasing in n, 
and we have 
ME, Loo уе 
im aeg le? 
Putting s — 0, we get 
А 1—P,(0)_,. Pr {X, > 0} 
= ILLIS] ———. 
9 (0) lim, m^ а т" (5.13) 


Let bj = Pr (X, = i| Xa > 0} and Bs) = È биз! be the p.g.f. 


of {bjn}. Then 
Різ) – P0) , _ 1 — Pa(s) 
BO TSAO -| T= P,(0) (6219) 
у Ler (РИЗ))/т 
=1- (1—3) П iron (from (5.12)). 
Since Р,(5) > P,(0), and r(s) is monotone decreasing, r(Px(s)) < r(P,(0)), 
and so each factor of the product on the r.h.s. expression is larger 
Thus &,(s) is monotone decreasing and tends to a limit B(s) 


than 1. 
as n > œ, i.e. 
B(s) = Ў b; s!, where bj = іш bj, = іт Pr (X, =j | Xn > 0). 
j=l noo noo 


ly B(0) = 0. 
Clearly B(0) B(P,(0)) = lim В4Р,(0)) 


a? — 1— PAPA 

=m {! TPO } 
20 f1— PAP (0) 

=i- lim {Seo | 


ү 1— Px(s) 
a) скы 


(since m < 1, Р„(0) > 1 as n оо) 
=1—m* 
(taking limit of (5.12) as s— 1). 


It follows that lim B(Px(0)) +1. As т< 1, Р,(0) = 1 as К-> oo. 


Sut wie Sacer 
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Hence B(s) - 1 ass+1. Thus B(s) is the p.g.f. of (bj. 
Further 


ә р -r 1 — В(Р,(0)) 
Ac BO) = =т= 
В тк 
т 
т“ 


К 1 
Tie PFS б}? Oy 
i 5.14 
Again, from (5.14) EEEO 
BAP) =) — TPO 
к= Paa) PO 
Sha). у= B 6-19) 


e 1 — Рь(5) : —1— 
Mm T= PL) m, (1— Ben) uh 


im l—P»4(0 ,, 1~P(s)_ 
LIVE arco ue т 


Hence taking limits of both sides of (5.15), we get 
B(P(s) = 1 — (1 — B(s))m 
= mB(s) + 1 — m. 
Thus the theorem is proved. A 
Remarks; The above simplified 
restriction is due to Joffe (1967). 
The equation (5.92) is known as a modified Schröder functional equa- 
tion. It is not always easy to obtain B(s) from it for given P(s). In 
Example 3(e) we obtained B(s) directly. 


The mean i КЬ, = 
kel 


proof which does not involve moment 


l/e(0) of the limiting distribution is finite iff 
E (X, log X) = А Pr {К log k} < © or p, — 1. 
=] 

Before closing the discussion о 


n G.W. processes we mention a few 
interesting innovations introduce 


d in the basic Process. 


Generalisations of the Classical Galton- Watson Process 

Branching processes with immigration: Theorem 9, 
а G.W. process, Pr {Xn > 0} = qand Pr {0 > k= 
so Pr (X, — œ} = 1 — 4), q being the probability of 
q=1 for critical and subcritical Processes. 
G.W. populations either di 
from outside into a criti 
effect on the Population size. 
itself is interesting from the 
Galton-Watson processes with 


4 states that for 
0, for finite К (and 
extinction. Further 
Thus left to themselves 
Immigration 
ve stabilising 
» immigration by 


| 
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cesses) often arise in applications in su isti 
cal mechanics, genetics, A ay ЭЙ Cr aL 

Consider a G.W. process with offspring distribution {рь} (havin Ў 
P(s) and mean P'(1) == т). (The process will be called nated mr OM. 
process.) Suppose that at time л, i.e. at the time of birth of sit vi 
tion there is an immigration of Y, objects into the population end 
Y„ n =0, 1,2, ... areiid random variables with p.g.f. i $ 


h(s) = r3 Pr (Y, =j} $ = У hjsi, 


i.e. with probability Aj, /immigrants enter thezth generation and contribut: 
to the next generation in the same way as others already present "a 
The numbers of immigrants into successive generations are inde ati: К 
and all objects reproduce independently of each other and of ‘he Has 
gration process. The distribution {Лу} will be called immigrant pond 
bution. Leta = // (1) be the mean of this distribution. 

Let Xn) be the number of objects at the nth generation and let 


Р (5) = z Pr (Xi =j} 5 
=0 


be its p.g.f. The sequence (Xy ? — 0, 1, 2, ...} defines a G.W.I pro- 
cess, The sequence is a Markov chain whose one-step transition pro- 


babilities are given by 
coeff. of s? in A(s) [Р], Є N. 


Pii 
Clearly, Peay (5) = MS) Po) (P). (5.16) 
It lim Pels) = F(5) exists, then one gets 
re 
(5.17) 


F(s) = h(s) F(P(S))- 
when it exists, satisfies the above functional equation. 


when does the limit exist, and when does it 
probability distribution or when does (Xi) 
Ü 


ie. the limit, 
Now the question arises, 
define the p g.f. ofa proper 
have a proper limit distribution ? 
Corresponding to Yaglom's result for subcritical G. W. processes, 
Heathcote (1965) obtained the following analogue for G.W.I. processes. 
Consider a Galton-Watson process with immigration, having offspring 
distribution (p;) (with р... PO) and immigrant distribution {/,} (with 
p.g.f. A(s))- If n<la= А1) < oo, then, lim Pr (Xo =j} = 4, 


exists for j € N, and the p. 


equation (5.17); 
Ез) = his) F(P(s))- 


у= X hj logj< o: 
j=) 


п-эсо 
g.f. К\з) E d; si satisfies the functional 


Further, Fl 
when 

1 — Bis) 
FS) apes 


It may be noted that, 
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1— P(s) 
and h(s) = т=з) 
then the functional equation (eqn. (5.9) which appears jn Yaglom’s 
theorem) 
B(P(s)) = mB(s) + 1 — m, 

reduces to the functional equation (5.17). Thus, ordinary subcritical 
G.W. processes may always be regarded as forming a subclass of G.W.I. 
processes (at least as regards Yaglom's conditional limit theorem). The 
result for the critical process, due to Seneta (1970), may.be stated as 
follows: "y | i 

For a Galton-Watson process with immigration having m = 1, o? < со, 
a< œ, the random variable X(n)/n converges in distribution, as n —> о, 
to а random variable having gamma density, 


u*-l е-щв 
теу, “> 


where æ = 2a/o?, B = с?/2. 


If a = (1) = 02/2, i.e. а = 1, then the gamma density reduces to an 
exponential density, such that 


lim Pr [e > uh exp {- = 
noo n a*J’ 


again from Theorem 9.8(c) we get 


lim {> u | СЕСЕ =); u 20, 
п->со п а 
for the G.W. process {Xn} without immigration. 

Thusin the critical case the effect of the Conditioning of non- 
{Xn > 0} is the same as immigrating into the process at an ay 
ofa= о?/2. 

Detailed studies оп processes w: 
several authors (e.g. Heathcote, 
book, one can find treatment of s 
above results, 


extinction 
erage rate 


ith immigration have been made by 
Seneta, Foster, Yang etc.). In Jagers' 
ome of them besides proof etc. of the 


Processes in Varying and Random Environments 
It has been assumed that the offspring distribution {pr} remains the same 
in all the generations and that objects reproduce independently of others. 
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chosen from a class of all reproduction laws and the i 

are said to be in random environments. The topic "dud ine 

great details in a number of papers by Wilkinson (1969), Athreya Wi 

Karlin (1970, 1971), Kaplan (1972) and others. › LU 
The models under these assumptions have wider scope for applica- 


tions. 


Multitype Galton-Watson Processes 

A natural extension of Galton-Watson process is concerned with the 
case where the population consists of a finite number of types of objects. 
cesses are known as multitype Galton- Watson processes. Mode 


Such pr 
d extensively with such processes, discusses 


(1971) deals exclusively an 
several models for applications in various areas. 

Suppose that a population of individuals (or objects) originates with 
a single ancestor and that there are k (finite) types of individuals. Let 
he probability that an individual of type i produ- 


р“) (ry г» ss n) bet 
ces rj offsprings of type / j=1,2,...,k. We introduce the vector 
notation: 


r = (My, P25 ++ +9 ry); 
8 = (Sp 2 -e 50) 
e; = (Bris Serr ++ +» 8н). 
р“) (т)? _.. st be the p.g.f. of (pOXr). Let 
X, = (Xa Х„®,..., Xa!) 


tion size of К types in the nth generation. 
ected number of offspring of type j produced by an 


Let f (8) = 


represent the popula 
Let туу be the exp 

object of type i. Then 
my = E (X | Xo = e) 


ef y is 
- a 265% 1), 1.2, cas Ke 
is the matrix of moments. Let f;/?(s) denote the 
bjects in nth generation starting from one object 
(2.5) is as follows: 


The matrix M — (mij) 
p.g.f. of the number of o 
of typei A result analogous to 


=f (2, (9), 79, (8); ---. f (9, (9), 


fa (8) 
fan) = 5 олке т ОТ 2,\.«'.: 
The population becomes extinct when X, = 0 for ѕоте п > 1. Let 
даб = Pr {X, =9 | Хо = е) 
and gi = lim e 
ie., 40 denotes the probability of до given that the process 
acestor of type i Let q— (4%, 4%), ..., 9M). 


arted with one а! ix of 
т uae D. the case where the matrix of moments is of 


a certain type. 


selves tO 
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Let M be a kx К matrix with non-negative elements such that for 
some positive integer п, all the elements of М" are Strictly positive. 
Then there exists a positive eigenvalue p of M, which is greater than the 
absolute value of any other eigenvalue of M; o is also called the spectral 
radius of M. И > 

When M is the matrix (тїр), this eigenvalue o in the multitype G.W. 
case plays the role of the mean m of offspring distribution in the simple 
G.W. case. A multitype G.W, pracess is said to be subcritical, critical 
or supercritical depending on o < 1, = 1, or > 1. We now state, without 
proof, the multitype analogue of' Theorem 9.3: 

Let the matrix of moments M be positively regular and let р be its 
spectral radius. 

(a) If p. 1, thengy— 1,1 = 1,2, ..., K, 

(b If р>], theng; < 1, i—1,2,..., К. 

(c) Ineither case, q = (q(9, g®, ..., 4%) 
is the smallest positive solution of the vector equation 

q = f(q). 
ie. 0 is the smallest positive root of qt? =f (q) for i— 1,2, ..., k. 
For proof and other results, the reader is referred 


to Athreya and Ney 
(1972) and Mode (1971). 


9.6 Continuous Time Markov Branching Processes 


So far we have been considering branching processes (X4) in discrete 
time; an object after one unit of time produces similar objects according 
to offspring distribution {px}. The lifetime of an object is one unit of 
time. Now we proceed to consider a generalisation such that the life- 
times of objects are i.i.d. random variables. Instead of the process 
UG, n = 0, 1, 2, ...} we shall consider the process {X(t),t > 0), where 
X(t) equals the number of Objects at time г. The process {X(t), t > 0) 
may or may not be Markovian. In this section we shall confine to the 
Markovian case such that Ou), 12 0} will be a Markov process in 
continuous time with discrete state Space. Such processes were conside- 
red in Sec. 4.5. We shall recall some of the basic results concerning 
such processes. We first note that the waiting times between changes of 
states (here lifetimes of objects) have independent exponential distribut- 
ions. Let the process be time homogeneous. 

Denote the transition probability by 

Pit) = Pr (tct —j| X() = i}, 
and, in particular 
рі) = Рг A) ex | X(0) = ij, j=0, 1, seal > 0. 
The transition probabilities Pit) satis 


fy the Chapman-Kolmogorov 
equation 


* Dit + T) = Be RIP Pe) 
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We shall assume here that X(0) = 1 i 
ly ie. th i i 
ancestor at time 0. * WAT p ДЕГ 
Let the probability that ап obj i i i 
i ject will split producing j obj i 
a small interval (t, t+ fi) be denoted by ajh + od» The buen 
о be independent of time) are the transition densities "E 


(assumed t 
d;— lim puh) j#1 
ho h 
a, = lim Pl) = zl 
so that 
py) =ajh + oh), i —0, phy ve: 


рп) = 14a hd o(h). (6.1) 


We have aj 2 0,]= 0 2, 3,...and 2 а; =0. 
ј=0 


Now objects act independently, and in one small interval, only one particle 


on the average, will split, so that 
Pr {X(t + /) =antk—1| XO= п} = nak h+ o(h), k — 0, 2, 3 
у=л | 0) == 1+ na h+ o(h). 


and Pr {X(t +h 
Let F(t, 5) be the p.g.f. of (p, (D)- We get 
2s js Bi E 
F(t, 5) = pits a Pr (X(t) =j | X0) =1} 5 (6.2) 
=E (s* o, 
and 
БО, 5) = $. (6.22) 
Definition. A Markov branching process {X(t), t > 0} is a continuous 
time Markov chain having for state space the set of non-negative integers 
М = (0, 1.2, „ыў and whose transition probabilities p;,(t) satisfy the 
relation 


L2, ps = 1806 905 i20,|s| <1. (6.3) 


оо 
X pj(0s-— 
Ј=0 

t characterises and distinguishes a Markov 


It is the property à e 
branching process from other continuous time Markov chains with state 
d in Sec. 4.5). The equation (6.3) is implied in the 


space N (discusse: 4 
postulate that individuals act independently. 
We shall now obtain a continuous time analogue of the relation (2.2) 


satisfied by а Galton-Watson process. 


(6.3) tha 


Theorem 9.10. We have 
re +T, 5 = [F(t, F(T, з), i > 0 (6.4) 
ке + DSA F(t, F(T, 5)). ae 


and in particular, 
Proof: Using the 
tion, we get 


relation (6-2) and the Chapman-Kolmogorov equa- 
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[50 + T, s)! = РА pilt Т) 5,120 


Z s {È рит) рку(Т)} 
Ј=0 kao 


3 pat) { Ў рТ) з) 
k=O j=0 


ll 


X Pal) (CT, 9) 
= [F(t, F(T, s), 

and, in particular, 

F(t + T, s) = F(t, F(T, 5)) 
We shall now proceed to obtain two differential equations (forward (6.6) 
and backward (6.7)), satisfied by F(t, s); these can be obtained from the 
Chapman-Kolmogorov forward and backward differential equations 
respectively. We shall, however, use a slightly different method. 


With the given initial conditions, the differential equations can be solved 
in terms of the transition densities aj. Let 


>> 7 
u(s) кл dj 5 (6.5) 
be the generating function of the sequence {a,}. 


Theorem 9.11. We have 


д д 
o F(t, 5) = u(s) а= F(t, s) (6.6) 
and È FU, з) = «Ft, з). (6.7) 
Proof: From (6.1) and (6.2) we have ' 


Fih, s) = Б рі!) s? 
-9 


(Ea hols + Е fah- od) s 
=s + ne 455) + o(h) = s + hu (s) + o(h). (6.8) 


Thus, using (6.4a), (6.8) and Taylor’s expansion for a function of two 
variables, we get 


Fit + h, s) = F(t, F(h, s)) 
= F(t, s + hu (s) + o(h)) 


= F(t, 5) + hu or: F(t, s) + oh). 


so that j ) - 
i F(t-4-h, s) — Fit, s) — КД TUE o(h) 
E Map m eS Т шз) == F(t, s) 4 imm "x 
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whence we get (6.6). Again, using (6.4a) and (6.8), we get 
Fh - ts) = F(A, F(t, 5)) 
TM = F(t, 5) - hu.F(t, 5))  o(A). 
ii = 
lim ще) лн = u(F(t. 8)) ; lim ott) 
[ L A» h 
whence we get (6.7). А á 
Each of the equations (6.6) and (6.7) is subject to the initial condition 
F(0, s) = s (eqn. (6.22). 
Solution of the equations. 
the forward equation (6.6) © 
of the initial condition БО, 5) = 5. 


F(t, s) can be obtained by solving either 
r the backward equation (6.7) with the help 


z 
Noting that 4 [Дд = f(z), and using F(0, s)= s, itcan be seen 
0 
(6.7) is equivalent to 
Ft) 
dx 
| А (6.9) 


"ON 


that the equation 


5 


) £0 in FO, з) =5<* <F(t,s). Thus F(t, s) can be 


Пу from (6.9). 
) together with F(0, s) = s leads to an identi- 


provided that u(x 
obtained theoretica 

The forward equation (6.6 
cal solution. 


Simple linear birth and death process. Here 


Example 6 (a)- 
Хи) = п) = пл o(h), k=2 


pr (x(t. n 5 k—1]| 
=n ph + ofh), КЫ 
21—n(Q4 wc o(h), k=1 
= o(h), k>3, 
e mere k23. 
Thus we have 
u(s) = as? — (A + e)s e 
From (6.9), WE have 
F(t, 5) F(t, 5) ЯХ А га 2 >. 
ИХ Е ИГ A Gi 
t= | Te © kx - DG - uM @=%) вар 
апа оп simplification, we get 
—1)-0s-— (pt 
(5—1) = OS u) е Tr 


F(t s= М5 = D-—0s— ш) е ` 
Expanding F(t s) as a power series in s, we can get p, (4) as coefficient 


266 STOCHASTIC PROCESSES 


of st, К = 0, 1, 2, ... [see, Sec. 4.4 under ‘Linear growth process’]. 
Probability of extinction: Tt may be noted that 


pit) = Pr {X(t) =0 | X(0) = 1) = FG, 0) 
is a non-decreasing function of /. The probability of extinction q is 
defined by 
gm pit). (6.11) 


Theorem 9.12. The probability of extinction q is the smallest root in 
[0, 1] of the equation u(s) — 0; further, g = 1 (q < 1) iff (1) < (>)0. 
Proof: From (6.7) we find that p,9(t) satisfies the equation 


d 
i iO = Gn). (6.12) 
Integrating, we have 
Prot) = f u( Pso()) dy, (6.13) 
0 
since Pio(0) = 0. 


We have u(1)=0 and u(0)>0. If u(0)=a,=0, then p,,(0) = 0 and 
q=0; then 0 is the smallest non-negative root of u(s) = 0, and the 
theorem holds trivially. Let u(0) = ao > 0 and g*.be the smallest root 
of the equation u(s) —0 in O<s<1, In other words, u(s) = 0 for 
s= q*, while u(s) > 0 in 0< s < 7*. Now p(t) is a non-decreasing 
function of ғ and so from (6.12) we find u( ps«(t)) > 0, for t > 0, i.e. 


A (PaO) is non-negative for all г> 0. Hence p(t) < q* for all t > 0. 
us 


q —lim p,«(f) < q*; 
LL 


further u(g)z 0. Suppose, if possible, that u(q ; 
RT , > 4) > 0, then whil 
lies in [0, 1], by the continuity of u(x), while pyo(t) 


t 


Ге) dy- оо as 1 оо; 
0 


аѕ lim Pxo(t) is finite, the above leads to a contradiction of (6. 13), which 


u(s) 


a 


Fig. 9.3 
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is due to the supposition that u(q) > 0. Hence we must have u(q) = 0. 
As q < q* and as q* is the smallest root of u(s) = 0, we must have q = q*. 
We have0« q—4* <1. Sinceu0) 2 0.4 —1 iff the function u(s) is a 
decreasing function at s= 1, so that и'(1) < 0, if it exists; on the other 
hand q < 1 iff /(1) >0. Hence the theorem. A 


u(s) 


Fig. 9.4 


Example 6(a) (contd.). In case of linear birth and death process 
u(s) == a (s—1) (8 EA Thus d= LiffA <u, ie iff (1) <0. 
9.7 Age Dependent Branching Process: 
Bellman-Harris process 

n we considered that the lifetimes of objects (or 
ЦЗ ; f tial random variables 
i dividuals, organisms) are exponente. $ s 
Cr aic en generalise this further; we shall consider that the lifetimes 
have general, and not necessarily exponential, distrib - 1008. 

а g 3 hal an object (ancestor) at time t =0 initiates the process. At 
d ppor its lifetime it produces а random number of direct descendants 
having offspring distribution {Ps} (with p.g.f. PG). We assume, as be- 
3 B hese descendants act independently of each other and that af 

"e M A ^ ERE descendant produces its own descendants with 
the end of i ү n Alistribution (pi), and that the process continues as 
the same 0 sp! c present. Suppose that the. lifetimes of objects are 
long as objects ar riables with d.f. G(which is also independent of the 
iid. random o Let (X(t > 0} be the number of objects alive 
offspring distributi cess {X(t), t > 0) is known as an age- 


A chastic pIO' à 
at time /. The ex time) branching process. Such a process is also 
ene 


dependent (or 8 p after Bellman and Harris who fi 
„Harris process, г "n 
known as à Bellman- in 1948. We shall consider here Bellman- 
considered such а dependent process. AD age-dependent process is, in 
Harris type age-C°F 
general, not Markovian. 


In the preceding sectio 
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Theorem 9.13 The generating function 


ка, з) = 5 Р(Х) =й) 5 (7-1) 


of an age-dependent branching process {X(t), t > 0), Хо = 1 satisfies the 


integral equation 
t 


F(t, s) = [1 — G(t)] s + | P(F(t — u, s)) dG(u). (7.2) 
0 
Proof: To find Pr (X(t) = К}, we shall condition on the lifetime T at 
which the ancestor dies bearing i offsprings. We have 


Pr (X(t) = К} fer {X(t) =k | T= u} dG(u) 
= fer (X(t) = k| T = ü} dG(u) 


& f Pr (X(t) = k | T = u) dG(u). 


In case of the second term, и > t. Given that T = и, the number of 
objects at time t is then still 1 (the ancestor) and the expression under 
square brackets equals 3,4. Thus the second term yields 8,, (1 — G(t)}. 

In case of the first term и < t, the ancestor dies at time u < t, leaving 
i (EN) direct descendants: the probability of this is pjdG(u); and further 
these i descendants (who independently initiate processes at time u) leave 
k objects in the remaining time t — и: the probability of this is equal to 
the coefficient of s* in the expansion of [F(t — u, 5)]; as a power series in 
s. Thus we have, 

Pr {X(t) = К} = (1 — G(0) 8, 


t 


+ | i P;dG(u) (coeff. of s* in the expansion of [F(t — u, s)]3. 
0 


Multiplying both sides by sk, k —0, 1, 2, ... and summing over k, we 
find that the l.h.s. equals F(t, 5); and that the first term on the r.h.s 
equals {1 — G(t)}s and the second term equals 

t 

| [2 rif 2, [coeff. of s* in (F(t — и, s)y] s*j] dG(u) 

-0 = 
e 
t 


Li 
= | [ 2 pi (ии, з) dal) = | P(F(t—u, з)) dG(u). 
0 0 
Hence the theorem. A 

Note: Theintegralequation (7.2) cannot easily be solved in the 
Beneral case. For such finer questions such as existence and uniqueness 
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of solutions of (7.2) and concerning properties of F, see Harris (1963) or 
Athreya and Ney (1972). 1 
Particular case: Exponential lifetime. Let GO=1—e*, then (7 2) 
yields ран 


t 
F(t, s) = se? + be~ Í P(F(v, s)) e dy 
0 


OF p P(F(t, 5)) — F(t, 5)]. 7:3 
дї (FC ( ( ) 


uivalent to the backward diftrohpat saut m (6.2) If 
then {Х@), t> 0) becomes a Markov. branching. process. 
Harris process with exponentially distributed lifetime 


which is eq 
Fi, 1) = 1, 
Thus, а Bellman- 


is a Markov branching process. T" 
It can also be seen that when G is a unit step function, (7.2) yields the 


functional iteration formula for P(s) in the discrete case, EN 
We now turn to the expectation M(t) == FAX} and its бата. 
behaviour. M(t) satisfies ап integra 


1 equation (7.4) which can be obtain- 
ed rigorously from (7.2). We shall Obtain (7. 4) directly by? üsing Argu- 
ments similar to those used to obtain (7.2). Further, we shall show that 
under certain conditions, M(t) is asymptotically exponential. 


Theorem 9.14. The expectation M() Е (Oy of an ;age-dependent 
process {X(t), t > 0}, Xo = А satisfies the integral equation? gaiU 


YG n5vig 


t 


M(t) = [1 -- G@)] + mf M(t — и) dG(u), (7.4) 
Di j'! s (o °з = (Yt 


: distribution. Further, if 
Suppose that iB the Unique 


Iswories үз Л 52 gnes vH 


where т=Р'(1) is the mean 
m= 1, then M(t) = Pisa so tion 
i 191081 


, 
positive root op" 


i fe | (7.5) 
d 2 (ууч = (3)M 7 
If m»! and G is not lattice, then as £ > o0, 
і 1 
| sss ЛЫ: - M PEPI E 
pirt M(t) > Heo i! | ^U) 
ani [nett dh аа vision 
0 
Proof: By conditioning on the lifetime T of the ancestor, we get 
o) RES » | 
ма) = [5 qr) UIT = u} 460) (7.7) 


о 
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If u >t then E(X(t) | T=u}=1. 
If u < t, then the ancestor lives for time u at the end of which, it leaves 
i direct descendants with probability рү; each of these direct descendants 


initiates a process, the number of objects of such a process having the 
same distribution as X(t — u). Thus for u < f 


E(X()| T= u}= i ipE(X(r—u) 


= {M(t — u)) m. 


Using the above argument we see at once that (7.7) yields (7.4). 


Obviously, when т = І, М(ї) = 1 is a solution of (7.4). Suppose 
that m > 1, and that « exists. Denote 


Ga(s) = т f e- dG(y); 


this implies that 
dGa(s) = m e-ss dG(s). (7.8) 
Multiplying (7.4) by e and writing v(t) = е-а M(t), we get 


V(t) = e= [1 — бүт] + Í v(t—u) аби), (7.9) 


which is a renewal type equation. 
Using Theorem 6.3 


У (Ch. 6) we find that it has the unique solution 
given by i 


Li 

WO = e (1—G(r) + [e {1 — Gt — д} ати), 
9 

where ma(u) is of the form indicated in the theorem. 


By using the Key renewal theorem (Theorem 6,7), we get, as t -> co 
! 


[ e™ {1 — G(t)} at 


е M(t) = убу se * 1. 


m T 


к= | udGa(u) = m Í ue" dG(u). 
0 о 
Integrating by parts the integral in the numerator, 


where 


we have 

t=o jt 3) 7 е d y 

ha © fe G(t) 
0 


1 1 қ 
a ^ (using (7. 5)). 


| e (1— GO} й = — {1 — a <= 


0 
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Thus, as t > со 
eM — — 7+. 
am? [ ue-*" dG(u) 


0 


Hence the theorem. A 
Note: A root « of the equation (7.5) 


1 

] e-?"dG(u) = "E 
0 
when it exists, is known as a Malthusian parameter ; the constant « repre- 
sents the geometric rate of increase postulated by Malthus (1766-1834). 
It is to be noted that when m > 1, there exists a unique positive root c; 
ifm-1,a-0. Ifm«l, there may or may not be a real root « and 
if it exists « < 0. 
Extincti6n probability is defined by 

q = Pr (Z(t) = 0. for all sufficiently large г). 
We state, without proof, the following theorem; for proof and for 
lurther results see Athreya and Ney ( 1972). 


Theorem 9.15. If the extinction probability 4 is 0, then F(s, 0) =0. If 
q>0, then for each f >0, Ft, 0)< % and (t, 0) is a non-decreasing 
function of t; further lim Ft, 0)—4 
a i; ]lman-Harris model it is assumed 
branching processes: In the Be Harris п me 
NU рга: offsprings at the end of its lifetime Theos pring 
"s lifetime. 

«tribution is independent of the mother's А М 
кооп ра real age-dependence was introduced in 1964 by 
Sevast’yanov, who introduced the possibility of dependence between 

2: у lif m c and offspring distribution. He considered the generali- 
mot "E m objet produces offsprings not necessarily at the end of 
sation t a get randomly chosen instants during its lifetime. For 
its lifetime bu Sevast'yanov type of branching processes, the reader 


results concerning 975) and Sevast'yanov (1971, in 
books by Jagers (1 : 
is referred to the 1968 papers): 


Russian; also to his 1964, 
isatii Continuous Time Processes 
TTE 4 Generalisations of oni 
Some Extensions ү ed conditioned limit laws of the G.W. processes. 
In Sec. 9.5, we cor һе section we indicated some of the extensions of 
y G.W. processes with immigration, 


Towards the end of t sscs, such as 
the classical G.W. Peot and random environments, multitype G.W. 
G.W. processes in Vary have also been done in the case of 


nsions E 
processes etc. Such er Continuous time multitype processes such 
d multitype age-dependent 


continuous time pog аа process an 
п om on studied in great details. Continuous time branching 
processes have ec! 
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processes in varying and random environments, processes with immigra- 
tion (Pakes, 1971, 1974, 1979), and processes with disasters (Kapian, 
1975) and others have received considerable attention. The study of the 
conditioned limit laws in various cases has been a topic of great interest. 
Functional equation ‘technique has led to new results in the theory of 
Jirina processes, viz. branching processes in discrete time ona continuous 
state space (Seneta). = nae ты 
Applications of branching processes have been made ‘in Several fields 
like biology, epidemiology, demography and also in nuclear physics 
and chemistry. The books by Mode, Jagers, Stinivasan, Keyfitz etc. 
discuss many interesting models of branching processos in such areas, 
/ НЕ covering theoretical developments.» «; nwon si >) i полу 
| Yt isnot) possible: to'give here references of even; ‘thgamost Ms toa 
мавз; ах vago number of references. given; here máy ‘be of «use toithe 
"ireadets.' 53d 10 WW.» J =~ wit 
We faye come a long way from the day when Francis: Galton; (1822- 
1911) first formulated the basic problem) of -branching, processes in 
Educational Times, London, Ist, April 1873, as under; 


'OPrgblem 4001: *A largé nation, of Whorri We will éonééin 'Gursélves ith adult níales, 
N in number, and who each bear separate surnames; colonise «a;district.;; Their law 
of population is such that, in each generation, ag per cent of the adult males have 

no male children who, reach adult life; ah have one such child, #2 haye, dwo, and, so 
on up to: ар, who have five. 

аера (1) What proportion of Surnames Will ауе! become Extinctiafter P génerations; 
and B, how many instances there will be-ofithe: same; штате! being held} Бууп 
persons’, el 


bom ишш 


111 rc wid Уууу хәл) 
ар "theory of brilchiag process was a foriaally initiated ‘in t é following 
bus by Galton and Watson (1874); 211 10 insbas 


9bni ei noitudimizib 
! Bránching* processes ‘have today “emerged ‘as\ al yt Пё field: of 


naan’ ‘both'from ‘the’point of view of theoretical "structures aiid ЫР 


М рын bros iie to several directions. fio bas smislil z'1 
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91 E {Хеып=0,1,2,...}, Xo 1l isa , branching, Process with 
B offspring р. 8; d PGs), show t "liat [PON r Ere х iW where k i js a 
i fixed. positive integer, is also a bran ih 


3 descendants, with probability i or q “respectively. Considering 
„ће root of the equation P(s) = = 5, verif hi t „the, prob 
~ ultimate extinction ‘is less than one 
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If the, offspring.distribution in.a,G.W. si i 
he, offspring.distribution 1n. 2,9: . process, 15, geomet k 
— 0, 1, 2, .,.,, show that PESSIMI dapi 


(дп — p") — (47% — p^) pas 


9.3 


bi 


Pri ae 

( п iai mug os phas ' prd А 

п (Шз 4 oes stb AL 
(п 1) ss? p 1. ae 


9.4 1.:Еіпа! the variance of. X, (i.e. the number of.objects in;nth.genera-; © 
: tion) ina G:W; process from the relation Pg Роа) 1o " 
95 Show that thevim.g.f. a(s) = Eten), mdi 2d satisfies the 

relations 


Фвы(т») = Р [0s(5)]. 
Show that the p.gf. of the conditional distribution of X; given 


X, 0is 


9.6 


P,(s) — P«(0) 
d JPEN Pa(0) ` 
г. cep LX. > 0} when;Pr (no. of offspring =e dy", 


97 Fora Galton-Watson (process, show that the expected, number of 


total progeny is given by 
E(Yn) = (== m'*)/(1 т), iml DDR 


Te Lahn 


ij 


asm- ©. — 


: e rA 
and that, if m< 1, EY) > [=m 


Show that | 
ovat (Ya) 3 dn En E1 mo¥/6, if m = ү! 
9.8 Show that:for-a: binary splitting process (i.e. with offspring distri- 


bution po,— 4» P2 7+ Р 1—q),ifp «i 
| — vAs*pq 


2sp 


\ 


ouHG) = 


(where H(s) Clm RS) Ra(s) = E GI» Y, — total progeny ир 
n9 


эп Jith generation). | Deduce that 
ob ati 10 Бш МЕ zr Odo. 


ирг {уш 24 = oe (4) pay. 


9.9 | Find the limiting distribution of the total progeny in a G.W. pro- 
cess with geometric offspring distribution {q*p}, k = (0s Ty Е 
9.10 Consider a G.W. process with geometric offspring distribution 
infgkpyy e 50; 1; Оу. Verify thatwhen:m = 1 
argia disob DAE і nil 2 ib Л 
lim n Pr {Xn > 0-3 
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9.12 


9.13 


9.14 


9.15 
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When m < 1 (9 < p), verify that Yaglom's theorem holds. Find 
b, = lim Pr(X,—k| X, > 0}, K=1,2,... 


п» ә 
and verify that B(s) = 5 bps" satisfies the equation 
к=] 
B(P(s)) = mB(s) + 1 — m. 
For a G.W. process with m < 1, c? < co, if B(s) is the the p.g.f. 


of the distribution {bp} defined by by (5 8), show that B"(1) < œ. 


Further, show that the variance o,? of (b) is given by 


Co? 
Wi Se = Г 
ge т — т? б 
where fus M be 
KS Г Ф (0) 


Time to extinction in a G.W. process. Show that the distribution 
of time to extinction T (which is the first passage time to state 0) 
is given by 


fn = Pr (T =n} = P,(0) — Р„_\(0), n= 0,2, ... 
Find ғ, when 


Q) ре= 9р, k = 0,1, 2,... 
(0) pk=(1— a(l — c) c, k= 1, 2, ... 
=a ,kz0. 


Suppose that at the end of exponential lifetime (having parameter 
4) each particle splits into k( 2 2) particles. Show that the 
generating function F(t, s) of {X(t) | X, = 1) is given by 


F(t, 5) = se% [| — (1 — e-at&-nn separe, 


Find var(X(t)), where X(t) is a Markov branching process with 
X(0) = 1 and having u(s) as the generating function of ће transi- 
tion densities. 

Consider an age-dependent process with exponential lifetime 
given by G(t)= 1 — e. Suppose that at the time of its death 
an object splits into two similar objects. The process then reduces 
to the pure birth process considered by Yule. Obtain the differ- 
ential equation for the generating function of the process. Solving 
the equation, show that 


Pr (X(f) = k} =e (1 — et, k—1,2, .... 


Starting with an age-dependent process show that the differential 
equation satisfied by a simple linear birth and death process 
reduces to 


— ——ü—  r€—À 
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Р АБ) F+ и, 


where 2, v. are the birth and death rates respectively. 
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CHAPTER 10 


STOCHASTIC PROCESSES IN 
QUEUEING AND RELIABILITY 


10.1 Queueing Systems: General Concepts 


The queueing theory had its origin in 1909, when A.K. Erlang (1878- 
1929) published his fundamental paper relating to the study of congestion 
in telephone traffic. Erlang is appropriately called the father of queueing 
theory (for his life and work, see Brockmeyer et al. (1948). The 
literature on the theory of queues and on the diverse areas of its appli- 
cations have grown tremendously over the years. The standard works 
and some of the other publications specifically referred to in our 
discussions are listed: in the references. For complete references, the 
standard works may be consulted. We consider here only the funda- 
mental aspects of the theory. 

A queue or waiting line (‘file d'attente’ in French and ‘Warteschlange’ 
in German) is formed when units (or customers, clients) needing some 
kind of service arrive at a service channel (or counter) which offers such 
facility. A queueing system can be described by the flow of units for 
service, forming or joining the queue, if service is not immediately 
available, and leaving the system after being served (or sometimes 
without being served). The basic features which characterise a system 
are; (i) the input, (ii) the service mechanism, (iii) the queue discipline 
and (iv) the number of service channels. 

By units, we mean those demanding service, e.g. customers at a bank 
counter or at a reservation counter, calls arriving at a telephone 
exchange, vehicular traffic at a traffic intersection, machines for repair 
before a repairman, airplanes waiting for take-off at a busy airport, 
merchandise waiting for shipment at a yard, computer programmes 
waiting to be run on a time-sharing basis etc. 

The input describes the manner in which units arrive and join the 
System. The system is called a delay or loss system depending on whether 
a unit who, on arrival, finds the service facility occupied, joins or leaves the 
system. The system may have either a limited or an unlimited capacity 
for holding units. The source from which the units come may be finite 
or infinite. A unit may arrive either singly or in a group. The interval 
between two consecutive arrivals (or between two consecutive arrival 
groups, in case of bulk arrivals) is called the interarrival time or interval. 

The service mechanism describes the manner in which service is 
rendered. A unit may be served either: singly 'orin a batch. The time 
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required for servicing a unit (or a group, in case of batch service) is 
called the service time. 

The queve discipline indicates the way in which the units form a queue 
and are served. The usual discipline is first come first served (FCFS) or 
first in first out (FIFO), though sometimes, other rules, such as, last come 
first served or random ordering before service are also adopted. 

The system may have a single channel or s-parallel channels for service. 
© The interarrival and service times may be deterministic or chance- 
dependent. The case when both the interarrival and service times are 
deterministic is trivial. We shall be generally concerned with chance- 
dependent interarrival and service times, and the theory will be essentially 
stochastic. When chance-dependent, the interarrival times between two 
consecutive arrivals are assumed to be i.i.d. random variables; the 


service times of units are also assumed to bei.i.d. random variables. 4 


Further the two distributions are also taken to be independent. 
The mean arrival rate, usually denoted by А, is the mean number of 


arrivals per unit time. Its reciprocal is the mean of the interarrival time’ 


distribution. The mean service rate, usually denoted by џ, is the mean 
number of units served per unit time, its reciprocal being the mean 
service time. In a single channel system, the ratio 


eae 
m 


is called the traffic intensity. Though dimensionless, it is expressed in 
erlangs {in honour of A.K. Erlang). The definition of traffic intensity is 


suitably modified. in case of s-channel systems and in case of bulk or 
batch queueing systems. 


Queueing Processes 

The following random variables or families of random variables which 
arise in the study provide important measures of performance and 
effectiveness of a stochastic queueing system. 

(1) The number Nr) in the system at time 7, i.e. the number at time t 
waiting in the queue including the one being served, if any. 

(2) The busy period which means the duration of the interval from 
the moment the service commences with arrival of an unit at an 
empty counter to the moment the server becomes free for the first 
time. 

(3) The waiting time in the queue (system), i.e. the duration of time 
a unit has to spendin the queue (system); also the waiting time 
Wn of the nth arrival. 

(4) The virtual waiting time W(t), i.e. the interval of time a unit 
would have to wait in the queue, were it to arrive at the 
instant t. . 

One needs to have their complete probabilistic description. It is clear 

that {M(z), t > 0), (W(t), t > 0}, {Wn, n > 0) are stochastic processes, the 
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first two being in continuous time and the third one in discrete time. 
It will be seen that some of the queueing processes that we would come 
across are Markovian and some are semi-Markovian. From some of 
the non-Markovian processes that arise, Markov chains can be extracted 
at suitable regeneration points and semi-Markov processes can be 
Constructed there from. The theory of Markov chains and semi-Markov 
Processes thus plays an importantrole in the study of queueing processes. 
Notation: A very convenient notation designed by Kendall (1951) to 

denote queueing systems has been universally accepted and used. It 
consists of a three-part descriptor A/B/C, where the first and the second 
Symbols denote the interarrival and service time distributions Tespec- 
tively, and the third denotes the number of channels or servers. A and 
B usually take one of the following symbols: 

M : for exponential (Markovian) distribution 

Ек: for Erlang-k distribution 

G : for arbitrary (General) distribution 

D : for fixed (Deterministic) interval 
Thus, by М/6/1 is meant a single channel queueing system having 
exponential interarrival time distribution and arbitrary (general) service 
time distribution. By M/G/1/R is meant the same system with the 
fourth descriptor R denoting that the system has a limited holding 
capacity R. 


Steady State Distribution 
N(t), the number in the system at time t and its probability distribu- 
tion, given by 


P«(t) = Pr {N(t) = n | №0) = +} 
are both time dependent. For a complete description of the queueing 
process we need consider transient or time-dependent solutions. It is 
often difficult to obtain .such solutions. Further, in many practical 
situations, one needs to know the behaviour in steady state, i.e. when 
the system reaches an equilibrium state, after being in operation for a 
pretty long time. 
Tt is easier and convenient to determiné 


Pn = lim p,(t) as t — oo. 


provided the limit exists. It is necessary to know the condition for the 
existence of the limit in the first place. This will be discussed in due 
Course. When the limit exists, it is said that the system has reached 
equilibrium or steady state and the problem then boils down to finding 
the steady state solutions. 


Some General Relationships in Queueing Theory 

There are certain useful statements and relationships in queueing theory 
Which hold under fairly general conditions. Though rigorous mathe- 
matical proofs of such relations are somewhat complicated, intuitive 
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and heuristic proofs are simple enough and have been known for long 
(Morse (1947)). It has been argued (Krakowski, (1973) also Stidham 
(1974) that conservation methods could very well be applied to supply 
proofs of some of these relations. Conservation principles have played 
a fundamental role in physical and engineering sciences as well as in 
economics etc. Similar principles may perhaps be applied in obtaining 
relations for queueing systems in steady state. Some such relations are 
given below. The most important one is 
L=,w (1.1) 

where à is the arrival rate, L is the expected number of units in the sys- 
tem and W is the expected waiting time in the system in steady state. 
A rigorous proof of the relation has been given by Little (1961) and so 
the relation is known as Little’s formula. This result, of great generality, 
is independent of the form of interarrival and service time distributions, 
and holds under some very general conditions. 

Denote the expected number in the queue and the expected waiting 
time in the queue in steady state by Lg and Wg respectively. These are 
related by a similar formula: 


Lg — AWg. (1.2) 


(see also Brumelle for a generalisation of (1.1)). 
A relation which holds for G/G/1 queue in steady state is 


A=(1—po)p or Po=1—p. (1.3) 


There are other relations, which hold under somewhat restrictive 
conditions. For example, for a system with Poisson input, the Pollaczek- 
Khinchine formula holds. We shall prove this formula later, while 
proof of Little’s formula will be omitted. 


10.2 The Queueing Model M/M/1: steady state behaviour 


This single server model envisages Poisson input and exponential service 
time with FCFS queue discipline. ‘Such a queue is also known as the 
simple queue (or Poisson queue or Markovian queue). 

The arrivals occur in accordance with a Poisson process with intensity 
^ (say) ie. the probability that an arrival occurs in an infinitesimale” 
interval. of length A is АЛ + o(h), while that of more than one arrival is 
o(h). This is equivalent to the statement that the distribution of the 
interarrival times is exponential having p.d.f. a(t) = 3e—*. "The. distri- 
bution of the service times is exponential, with parameter p (say) having 
p.d.f. b(t) = pe. In other words, the probability that one service is 
completed in an interval of infinitesimal length: is zh + o(h), while.the 
probability of more than one completion of service is o(h). It is assumed 
that interarrival times as well as service times are stochastically indepen- 
dent. We shall call such a queue an М/М/1 queue with parameters?, ш 
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(or a simple queue with parameters à, y). The mean interarrival time is 
1/A and the mean service time is | lu. The ratio р = А/и іѕ the traffic 
intensity (also called load factor or utilization factor). 

Let N(t) be the number in the system (those in the queue including the 
one being served, if any) at instant t > 0; then {N(t), t> 0) is a Markov 
process in continuous time with denumerable number of states A 97 
Here transitions take place only to two neighbouring states. This isa 
type of birth and death process. 

The arrivals here сап be thought of as “births” and the service comple- 
tions as “deaths”. The birth and death process with А, = A, n > 0 апа 
Un =p, n > l, us = 0, is also known as: immigration-emigration process 
(see Sec. 4.4). Let Pr{N(t) п | №0) —-)-—p.(t), n 2 0. Using the 
same probabilistic arguments (or directly from equations (4.6) and (4.4) 
of Sec. 4.4) we at once get 


Do (t) = — Ape(t) + vp, (2) (2.1) 
рт (f) =—(A +B) Pat) + APoa(t) + ира), n>1. (2.2) 


A. Steady-State Solution 
First, let us assume the existence of a *'steady-state". Then as t tends 
to infinity, p,(t) tends to a limit pn, independent of t. Theequations of 
steady-state probabilities p; can be obtained by putting p,'(f) = 0 and 
and replacing p,(t) by p, in (2.1) and (2.2). We get 
0 = — Apo + up (2.3) 
0 — — (A + ш) ра + Apna + рт, п = 1,2,.... (2.4) 
Dividing by u, and writing р for A/u, the above equations can be 
written as 
Pı — еро = 0 (2.3a) 
Pria — (1 + 9)Pn + PPn = 0, n— 1,2, .... (2.42) 
The solution of these equations was considered in Example 5(b) Ch. 1. 
The solution is given by 
Рп = рор", п> 0. (2.5) 
Now іп order that {рь} is a probability distribution, we must have 


X pn =1 or po Ў р" = 1. 
n=0 n=0 


The infinite geometric series converges to the sum 1/(1 — р) iff o < 1. 
For existence of steady state solution, р must be less than.1. We have 
then 

Ра = (1 — 9) е", n> 0. (2.6) 
The distribution {p,} of the random variable N, the number of the 
system in steady staté, is geometric (see also Example 4(c) Ch. 7, 
where the distribution was obtained by using the theory of semi-Markov 
processes). 
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The probability that the system is empty is ро = 1 — р and that the 
system is not empty is given by Pr {N 21) —1—p,—1—(1—9p)— р. 
We have 


=== g pee 

PIT p pt 
and Var ese e en Mes 
(1—0)? (ы = А)? 


The following table gives values of E{N}, var (N) for some values of р: 
e 0.1 | 0.3 0.5 0.7 0.8 0.9 0.95 0.99 
E(N) 0.11 043 1.00 2.33 4.00 9.00 19.00 99.00 
var(N) 0.12 0.61 2.00 7.78 20.00 90.00 380.00 9900.00 


Note: р < 1 (or equivalently ^ < р.) is a necessary condition for the 
existence of steady state, and that the queue discipline has no effect on 
the distribution of N. The queue discipline will have to be taken account 
of in obtaining the following distributions. 


Waiting Time Distributions 


Waiting Time in the Queue 

We assume that the queue discipline is FCFS. Let the random variable 
Wo denote the time spent waiting in the queue (or ‘queueing time’) of a test 
unit. Now Wg=0 when there is no unit in the system, the probability 
for which is p, — 1 — р. Given that there аге n(>1) units in the 
system, Ио = S,, where 


S, = Vy + Vet oes t Ул, 


v', being the residual service time of the customer being served and 
Yo Va» «++» Уп being the service times of the units in the queue. Now v's, 
the residual of an exponential distribution with mean l/u is again an 
exponential distribution with the same mean, Thus S,is the sum of 
n i.i.d. exponential distributions and is a variable with gamma distribut- ` 
ion having p.d .f. 
u” x?71 етых 
T x 230 
(see Theorem 1.10). 
Hence we have, for x > 0, 


w(x) dx = Pr (x < Wg < x + dx) 
= A Pr (x < S, < x + dx | the test unit finds п in the system) 
"ES 
- *Pr (the test unit finds п in the system) 


оо п xn- e-ux 
lem 2 ахі ps 


T'(n) 


n-i 
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со ох n—1 
= ae {5 = = ^ 


= up (1 — p) ect O70 dx, x > 0. 


"Thus the p.d.f. of the distribution is given by 
w(x) = 0, x=0 
= us (1 — р) е-®(ї—р)х, x > 0. (2.7) 
The distribution function F(x) of Wo is given by 
F(x)=1—9,x=0 
x 
and F(x) = io dt 
0 
= pea 0; (2.8) 
The above gives the (unconditional) distribution of the waiting time of 


a test unit. The probability that the unit has not to wait is 1—p and 
that it has to wait is p. We have 


E(Wg) = 0: (1— в) + ] рр (1 — р)х е#0-Р)х dx 
0+ 
2 A 
= Á—P—G" 2.9 
(1—0) e(e— А) тз) 
Now the (conditional) distribution of the waiting time in the queue 
denoted by ¿Wọ given that the test unit has to wait, is given by 


f(x) dx = Pr (x < Wo < х + ах), х> 0 
= Pr {x < Wo < x + dx | the test unit has to wait} 


— о) e-ua-o. 

= — dx = y(1 — р) eae dx, x> 0. 

The distribution of „Жо is exponential (with mean 1/p (1 — р)) while that 
* of Wg is modified exponential. 


Waiting Time in the System 

The random variable *time spent in the system' (or sojourn time) by a 
unit includes the service time of the unit besides its queueing time. This 
variable is given Ws = Wo + S, where Wo is the queueing time and 5 is 
the service time, and its distribution can be obtained by convolution. We 
give below a derivation similar to the one given for Wo. 

Now the test unit has to wait in the system even if the system is empty 
(п = 0), the time spent being equal to his service time. Given that there 
are n(7 0) units in the system, its waiting time is Ws = Sp4,, where 
Snapa = Y, + Vy +... + Улы» Vapa being its own service time. 
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Thus the p.d.f. w,(x) of Ws for x > 015 given by 
w(x) dx = Pr (x < Ws < x + dx} 


= Pe {x < Sry < x -+ dx | given that 
n=0 
there are n units in the system): Pr {N = п). 


={ 3 пена ое) 
= y (1— 6) ecet-e dx, (2.10) 


Thus Ws has an exponential distribution with mean 1/{0(1 — р)) = 
1/(Ł — à) and variance 1/(v. (1 — 6). 

The distribution function is 

F(x) = Pr (Ws & x) == f w(x)dx = 1 — е—®Ч—р)х, x > 0. 
0 

Note: (1) S44, is the random (geometric) sum of (n+ 1) i.i.d. 
exponential variables. That its distribution will be again exponential 
was also shown in Example 3(g) Ch. 1. 

(2) Itcan be easily verified that Little's formula holds. 


Example 2(a). The arrivals at a counter in a bank occur in accordance 
with a Poisson process at an average rate of 8 per hour. The duration 
of service of a customer has exponential distribution with a mean of 6 
minutes. Find the probability that an arriving customer: (i) has to wait 
on arrival, (ii) finds 4 customers in the queue, and (iii) has to spend less 
than 15 minutes in the bank. Estimate also the fraction of the total time 
that the counter is busy. 


Here A= 8/br., р = 10/һг., p=0.8. 


The probability that the customer (i) has to wait is given by 
Pr {N > 1) = р = 0.8, and (ii) finds 4 customers is given by p, = (1 — р)“ 
= (0.2) (0.8)4 = 0.08192. 

The probability that he has to wait less than 15 minutes (= 0.25 hr.) 
is given by 


F(0.25) = 1 — exp {— u (1 — p) (0.25)} (from 2.10) 
= 1 — exp ( — 10(0.2) (0.25)) = 1 — e-95 = 0.3935. 


The utilization factor р =: 0.8 gives an estimate of the fraction of the 
total time that the counter is busy. 


B. Queue with Limited Waiting Space: The queueing model-M/M/1/R 

In the M/M/1 model, it is assumed that the system'can accommodate 
any number of units. In practice this may seldom be the case. We have 
thus to consider the situation such that the system has limited waiting space 
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and can hold a maximum number of R units (including the one being 
served). The analysis is similar. Equation (2.1) will hold but (2.2) will 
hold only for 1<п< (А — 1). Further, there will be another relation. 
Suppose that the number of units in the system in an interval (0, t) is R, 
then the probability that there will be R units in the interval (0, г + A) is 
given by 
pa(t + №) = pr(t) (1 — uh) + pai (АЮ) (1 — wh) + off) 
which gives, as л 0 
pr'(t) = — u pa(t) +à pr-a(t). 
Thus the steady state equations are 
0= pot uP 
0 — — (A + p) ра +À Pait E Pr п 1,2,...,К — 1 
0=— u pr +À pg (2.11) 
From the first two equations we get, asin (2.5) 
Pr = ро 0, 0S n< К—1. 
From the last one, pr = ppr-1 = por, so that 
ps роб" for п = 0, 1, ..., R. 
в 
Using У pn =l; we get 


n=0 


= RET? pel. 
Finally, we have, for n = 0, 1, ..., R, 
(1—9e" 
Pn = “peer РУ 
1 
= = 212 
г аа ae 


The distribution is truncated geometric for р # 1 and uniform for p= 1. 
The mean number of units in the system Lg is given by 


R 
Ір= Х пр». 
ne 
| R(R+1/2 К 
For p = 1, Ly - 02 
1— R 
For e #1, Ip = GOES ner 
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ML=Oie «d: 2 J 

te oe e 
Сре 1. (АЧЫЛ) eRe RoR 
leat (1 — р)? 


(2.13) 


R 
It is to be noted that since the finite series X c" has a sum forall values 
n=O 


of р (2 < land e > 1), the steady state solution exists even for р> 1. 
Some of the results for M/M/1 model (or М/М/1/со model) can be 
obtained by taking limits of the corresponding expressions as R > co. 
But then this could be done only when р < 1. 


10.3 Transient Behaviour of M/M]/1 Model 


We have now to solve the equations (2.1) and (2.2) in transient state. 
A number of methods have been put forward for their solutions: the 
spectral method of Lederman and Reuter (1956); the method of gene- 
rating function of Bailey (1956): the combinatorial method of Champer- 
nowne (1956). The methods are given in all advanced level books on 
Queueing theory (e.g. Prabhu, Saaty, etc.). We consider Conolly's (1958) 
difference equation technique and also outline Bailey’s (1956) generating 
function method. 


Difference Equation Technique 

For obtaining the solution in steady state the number in the state at 
the initial stage is of no consequence. Неге the knowledge of the initial 
state is needed. 

We assume, for simplicity, that the number of units at the instant 
t= O 150. Let 


Palt) = Pr {N(t)  n|N(0) = 0}, рь(0) = 1, pa(0) = 0, n #0. 


We recall the approach to solution of the equations (2.1) and (2.2) as 
considered in Example 6(c) Ch. 1. 
Let f,(s) be the L.T. of p,{t). Then from (2.1) and (2.2), we get 


(s+) fols) = 1 + wh(s) (3.1) 

- (S EA + a) fals) = Mai) + (5), n> 1. (3.2) 
The solution of the equation (3.2) is 

fs) Ax” i ВВ", n z 1, (3.3) 


where А, B are functions of s (and not of n) and o, B аге the roots of 
the characteristic equation 


ux*—(s:A-gug)x FA—0. (3.4) 


We are now to find the values of А and B. 


STOCHASTIC PROCESSES IN” QUEUEING AND RELIABILITY 287 
Writing K(s) = K = {(s А + p)?—4ay}”, we get 


= блюз («= a(s). В = B). (3.42) 


It can be seen that, for all | | and real and positive ^; g, 


|«|» 1 and |8| < ! (see note below). 


Now B p(t) = | and hence 5 fes) )=1. 


Since | |> l, the convergence of b dy (Аал + AB") 
n=0 n»0 


implies that 4 = 0. Thus f,(s) = B8", n > 1. Choose B such that this 
holds for n = 0 also. 


We get Jo(s) = B, so that 
fals) = Љ(5) В", n > 0. (3.5) 
Again, fg) = ЕВА). BAG) 5 + 0/65) 8 (using (3.1) & (3.5) 
whence, using up? - (s+ A-+ pu) B- А = 0, we get 
l LG 


AG) — ise s 
Thus fals) =fo(s) 8" = Ss (3.5a) 


P,(t) can be obtained by taking the inverse Laplace transform of f,(s). 
This is done by using some properties gf Laplace transforms and by 
taking recourse to ingenious and somewhat complicated algebraic mani- 
Pulations (see Gross and Harris, for details). We proceed to find the 
Probability Pr (N(t) > nj. From (3 5a) we get 


BS pr IRR ame 


žno 05 na a ү 
= Qu)" «—"/5, (3.6) 
= L{A,(t)}, п> 0, 
where A,(t) = Pr (N(t) 2 n) = pat) + Pnsilt) + ++ 


The function A,(t) can be obtained by finding the inverse L.T. of (3.6). 
From the tables of L.T., we find that the L.T. of f; nan} is 


a y 
з + 4(*—a)' 
Using the translation property, the L.T. of 
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а) = ( 2)" БЕУ, fe 1,0214) } (3.7) 


is found to be 
(cr Era] - 


1 
and hence the L.T. of f a,(u) du 
0 


is Qu)" a/s. 
Thus the derivative of A,(t) is a,(t), given by (3.7) (A, is the distribu- 
tion with density a,). s 
The steady state probability p, — lim Prt) can be obtained by 
t>% 
taking limit of Ppt) obtained after inverting f,(s). It can also be 
obtained easily by using the initial value theorem of Laplace transform. 


Now lim В = lim @(s) = lim IAS US VAG + аа EN 
зво 5 $30 290 2u 


CENE, when à < p 
E (3.8) 
a+ ною. 1, when à > y. 


Thus p,— lim p,(t) = lim 5/,(5) 
[300 $0 
= lim (1 — p) gr 
4-0 


-r —р)р”, р < 1 
0 T» 


The result p, =0, p > 1 may be interpreted by saying that when the 
traffic intensity is greater than or equal to 1, there is ultimately zero 
probability that the system Contains a finite number of units. One would 
intuitively expect that the queue size would grow larger and larger in 
such cases. 

Another fact to be noted is that, when e = l, the states аге persistent 
and null having infinite recurrence time. - 


Method of Generating Functions 
To cover the more general case, let us suppose that МО) = i, ie. the 
process starts with i units at time г = 0; and we denote 


Pr (N(t) = n | №0) = i) by Pin(t) so that p,,(0) = 8 


(so far we assumed that N(0) = 0; putting i= 0 and writing p,(t) for 
Pin(t) when i=0 we can get back to the case last Studied). For the 
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difference equations (2.1) and (2.2) we shall now get 


Piolt) = — Aput) + рї) (3.9) 
Pilt) = — (A + u) pin(t) + Apinca(t) + Pimat), M=1,2,.... (3.10) 
Let P(A) == Ў Pin(t) 2" 


be the generating function of {pin(t)}. Then P(z, 0) = zi. Let 
Sin(s) = L { Din(t)} 
F(z, s) = L{P(z, t)}. 


Multiplying (3.10) by z^ and adding for n= 1, 2, ..., the resulting 
equations with (3.9) we get 


and 


„5 Pint) 2" = — (A + p) X Pint) 2" + ырк!) 
+A i Pin-,(t) 2" -- u 2, Ріп+1(1) z^ 
ne ne 


en эк PG, t) = — (+ p) Plz. t) + ири) + AzP(z, t) 

É + (6/2) tP(, t) — pu (t) 
or MT P(z, t) = {№2 — (à | u)z + Ы) P(z, t) — u (1 — 2) po(t). (3.11) 
Taking L.T. and noting that 


А E P(z, о} = sF(z, s) — P(z, 0) 


= sF(z, s) — гі, 
we get 


SzF(z, s) — z = faz? (А + p) z + Ы) F(z, s) — w(1— 2) fials) 
or F(z, з) = ЩІ — 2) fials) — 2+! 12 
(2, 5) WARN 6.19 


This involves Ло(5) which can be evaluated as follows. The denominator 
of the r.h.s. has two zeros, say, Ё and n, 


Where Ene t^ E aeu 


further, |E| « 1, |?|2 l. Note that the roots of the equation 
Az? (фр 4-s)z4 в == 0 are the reciprocals of the roots of the 
equation (3.4). Thus Ё, таге the reciprocals of a, B, ie. E =I/a, 
7 = 1/8, and thus |&Ё|<1 and |7|>1. It can be shown that 
Az? — (A + p s)z +u = 0 has only one zeroz =€ in the unit circle. 
Since F(z, s) converges in the region |z| = І, the zeros in the unit circle 
of the numerator and the denominator of the r.h.s. of (3 12) must 
Coincide, so that z — E must be a zero of the numerator also. Hence 


Fiols) = E*'/u(1-— E) 


"TO Lg рае 
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and finally " 
(1 E z) +1 — 21+1(1 —— E) 
ACL — 5)(2 — 8)(2 — у)" 
Expanding the r.h.s. of (3.13) іп powers of 2, fin(s) сап be obtained as 
the coefficient of z^. 
Particular case: When i = 0, i.e., №0) = 0, we get 


F(z, 5) = (3.13) 


Кака a “x= TEXTE =) d 
ес en CP Ege 
(since &(« — 1)(8 — 1) = — s). 
Hence, (5) = fon(s) 
= coeff. of 2" іп F(z, s) 
(1 — 6)" 


ss in (3.5a). 

Note: That|x| > 1 and |$] < 1 can be best demonstrated by applying: 
Rouche's Theorem. If f(z) and g(z) are functions analytic inside and on 
a closed contour C, and if | g(z) | < | f(z)| on C, then f(z) and f(z) + g(z) 
have the same number of zero (s) inside C. 

(For a proof of this theorem, reference may be made to any standard 
book on Complex-variable theory.) 

Here we take C to be |z|=1, f(z)=(A } u+ s)z, 8(2) =A + uz 
and Re(s)>0. We have, on |z| == 1, 


0) | = 1 uz! p<[statpl=|(s+r+y)z]/=/fe)| 


Now /(2) = (5 4А Б ш) 2 has only one zero inside 12| = 1 and so 
uz? +A—(s+ +p) 2 = 0 also has only one root inside |z| — 1. This 
equation has two roots х, of which В has the smaller modulus. Thus 
[В|< 1апа|е|> 1. 

A more sophisticated appro^-'i not involving generating function and 
Rouche's theorem has been Tecently given by Neuts (1979). 


Busy Period : Zero-avoiding state рғораришех 

A busy period is defined as the interval of time commencing at the 
instant 0 when a unit arrives at an empty counter and terminating at the 
instant when the server becomes free for the first time. Let the length of 
the interval which is a random variable be T, and (N*(r)) be the stochastic 
process denoting the number of units present at the instant г during the 


busy period. We have N*(0) = 1 and the duration of the busy period 
is the first / for which N*(t)=0. Let 


ant) = Pr (N* (1) = n| N*(0) = 1}, (3.15) 
be the zero-avoiding state probabilities. 
Then 4100) = 1, 440 —0, "22. For n22, д4) will satisfy the 
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same differential equations as p«(t), i.e. the equations (2.2) will hold also 
for gn(t) forn = 2,3, ..., ie. 


qa (t) = — (ù = ш) galt) + Adult) + uqaa(t). (3.16) 


The equations corresponding ton = 1 and n — 0 will be as follows: 
а'@) = A ` ш) a(t) uq (3.17) 
qo (t) = Balt). | (3.18) 

Let ga(s) be the L.T. of q.(f). Then taking L.T. of (3.16) we get, for 


n>2, 
ses) — 4.00) == — (A Бш) gu) + Mri) + шпа (5) 


or Ugna) — (S +), ш) gas) + 5ёл—1(5) = 0. 
The solution of this equation is (as in (3.3)) 
g(s) = Ax" > ВВ", n 22, [a] >I, [I< 1. 
We can choose the constants A and B so that they yield correct values 
of g,(s) and go(s). Again. PECQI must converge so that 4 is equal to 


zero, Thus putting n 1, we get g,(s) = BP so that 
gals) = (0) BY, n 1. 3:19) 
Taking L.T. of (3.17) we find 
s g(s) — (0) = = (A = u) mG) + BB e), 


- £ (as B satisfies (3.4)). 


1 
whence a(S) = тайкы uB 
Ts qs) = gs) B7 = ВИА plot, п =1,2,.... (3.20) 
Inverting the L.T., we get, form = 1. 2.. .. . 
nj a „э 
GA) = cS se QR) € L(2t ҳу A). (3.21) 


Let b(t) be the p.d.f. of the busy period T. Then, for small èr, 
b(t) òt =Pr{t<T<t + èt} 


equals the probability that the busy period terminates in (f, t + ôt). It is 
Clear that this probability equals the probability that there are j ( > 1) units 
in the system at the instant г and that the services of these j units are 


Completed in (t, t + 8). Thus 
ble) èt = È [Pr (N*(0 =j į N*0) = 1} 
jel 
-Pr {j units complete service in (t, t + 30] 


= q(t) e òt 5 90 0008) 
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so that b(t) = p a(t) 


= I ert eon 1, (2t Xj). (3.22) 


(using (3.21)). 
The L.T. of the p.d.f. of T is 


b*(s) = 1 {(t)} = Цы q(0) = u g(s) = Bio. (3.23) 
Using (3.8) we get, 
Pr {T< c0) — lim b*(s) = 1, if ecl 
s>0+ 
Me, df Р 21, 


indicating that there is non-zero probability that the busy period lasts for 
ever when p> І. 


We now find the moments of Т. We have for р<1 


BEOS 0-264313: 2k) — Ё, (к= КО) 


а 2 
2з В @ =» AG rate = 


Thus E(T) = -2 b*(s) T Al. = 5 = =e) 
(3.24) 
ar) = BOL =P р ag 329 
and var(T) = eo 
The coefficient of variation Cy = CoA » 100 
CETE 


increases as p increases, while it is constant for Ws. The following table 
gives E(T) and var (T) for some values of 4, р (и = 1, A = р). 


^(—80 02. 0.4 10,5 “оз 0.9 0.95 0,995 
ET) 1.25 1.67 2 5 10 20 200 


var(T) 2.34 6.48 12 225 1900 15600 1596 «108 


It is clear that Cy is large for large p, i.e. var(T) is large compared to 
E(T); as such the expected value does not seem to have much significance 
for large p. 

Note: (1) Idle (or free period) is the duration of the interval from the 
instant the server becomes free for the first time to the instant of the next 
arrival when he becomes engaged. This duration being the residual 
portion of the interarrival time is again exponential with the same mean 
1/^, since the interarrival distribution is €xponential with mean 1/4. 
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(2) We have considered the initial busy period, ie. the period 
initiated by one unit. The distribution ef the busy period ү, initiated by 


j units is given by the convolution ту Р: To where T; are i.i.d. random 


variables distributed as T considered above. 

(3) The concept of busy period is due the French mathematician 
Emil Borel (1871-1956), who obtained the joint bivariate distribution of 
T and N, the number served during the busy period T for M/D/1 model. 
Various approaches are available for obtaining the distribution of T as 
well as the joint distribution of 7 and N in more general cases 
Distributions of T and N can be obtained as marginal distributions of the 
joint distribution. Thetrivariate distribution of 7, N and M, where M 
denotes the maximum queue length attained during the busy period T, 
has also been engaging attention of researchers. 


Example 3(a). First passage time distribution. For а M/M/1 queue in 
transient state, let + be the random variable denoting the time taken for 
the system to fall from the initial state a to another state b( < а) for the 
first time. Let X(t) be the process denoting such states of the system, 
given X(0) =a, 

q«(t) = Pr {X(t) = n | X(0)— а}. (3.26) 
Using similar arguments it can be easily shown that the state probabilities 
satisfy the following equations: 

qa (0) = — (A+ 9) qi(t) + Agn-a(t) + р4һ+(@), n 2 (b + 2) 

44100) = — (A+ ш) gest) + uqoss(t) 

qe (t) == uit). (3.27) 
The p.d.f. f(t) of the first passage time distribution from a to b is given 
by f(t) = рь+1(1) = 9 (0). 

Putting a—1, b= 0, we can get back to initial busy period distribution. 
Waiting time process. Let W(t) be the time required to serve all the 
units present in the system at the instant /, given a. W(0)—0. Then, 
if N(t) is the number present at instant г, (N(0) = 
if e = 
Жие =f% +» beset Yms Gf N > à 
where v,’ is the residual service time of the unit being served at the 
instant /, and vs, ...,vw() those of the waiting units at the instant t. 
{W (t), t > 0), which is known as virtual waiting time, is a Markov process 
(with continuous state space). Given W(0) = 0, then proceeding as in the 
case of waiting time in the system in steady state it can be seen that its 
probability element f(x, t)dx = Pr (x < W(t) < x+ dx); 0x x « о, 
0 « t < oo is given by 
f (xy 


Дх, 1) dx = E CEN +1) 


етв dx} Pnt)» 
0 
Its L.T. can be put in elegant closed forms (see Prabhu (1965)). 
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10.4 Birth and Death Processes in Queueing Theory: 
Multichannel Models 


A. Birth and Death Processes 

Let us first consider a birth and death process with state dependent birth 
and death rates àn and un respectively. Let №(/) be the number present 
at the instant г, and 


Palt) = Pr {N(t) =n | N(O) - -} 


It was shown in Ch. 4 that (N(r), t > 0) is a Markov process with denu- 
merable state space (0,1,2,. . . and that the forward Kolmogorov equations 
of the process are: 


Po (0) =- — Арыг) + Фр) 
Рп (I) = -(%а-+ ил) Palt) - capaci) + Ил+1Ррп+1(1), п =1,2,... 


(see equations (4.4) апа (4.6), Ch. 4). 
We proceed to investigate the steady state solutions. Assume that such 


solutions exist, then lim p,(r) = p, = Pr {N =n}, N being the random 
too 


variable giving the number of units. Putting рї) pa and Pn'(t) =: 0, 
we get from the above the following difference equations in steady state: 


0= =Po лр, (4.1) 
Ü s» — (An E Un) Dart Aaa Pag Baia Prag 591,2, os vv (4.2) 
From (4.2), we have for n = 1, PENES 
Uni Poti — Àn Pn = ра рп — An, Pai 
= Ma-iPa-i — Ang Pn- (putting n — 1 for n) 
= nn d ^o Po P 
=0 (from (4.1)), 


so that Dna = 


= (>) ( = 
Una Un Pie 
ТЕО Ao 
nyi Шт." 


Е Е п 
ог pa =.) m po n—1,2, .... (4.3) 


Since X p, must be unity, 
n=O 


c n i 
i Sie i 
{ пее Uk Рот 1 ' (4.4) 


Hence a necessary and sufficient condition for the existence of a steadv 
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state is the convergence of the infinite series zn з (Karlin and 
п=1 k=] Uk 
McGregor (1957)). When it converges, po can be found from (4.4). 

The queueing processes arising in some of the standard models can be 
considered as birth and death processes and the steady state solutions 
can be obtained easily by using the above. In the M/M/1 model con- 
sidered in Sec. 10.2, 44 = n = 0,1, 2, ..., un = п = 1, 2, .... Putting 
these values in (4.3) and (4.4) we at once get the steady state solutions. 


B. The Model M/M/s 

Here we consider a queueing model with 5(1 < sx оо) servers or 
channels in parallel and having identical input and service time distri- 
butions as in the model (M/M/1). In other words, the present. model 
(М]М]$) considers a-Poisson process with parameter à as its input 
process and has, for each of the s channels, iid. exponential service 
time distribution with mean rate џ. If n(X s) channels are busy, the 
number of services completed in the whole system is a Poisson process 
with mean ny and thus the time between two successive service comple- 
tions is exponential with mean I/nz; whereas if n(> s) channels are 
busy, the time between two successive service completions is expo- 
if N(t) is the number present in the system at 


nential with mean 1/5и. 
densities are as follows: 


the instant г, the transition 
anny £ Pr {Nt + dt) = п= ПМ) == adt — o(dt) 
акал = Pr{N(t + dt) =" — 1| NU) = n} = padt + o(dt) 
nym = Pr{N(t + dt) = тім) = п} = о(@), m#n—l, n= 1 
where Ua = Mes if O<n<s (4.5) 
=s. if ons 
and death process with constant arrival (birth) rate 
service (death) rates as given in (4.5). 
N(0) = +} and let the steady state solutions 


Thus Mt) is a birth 

An = ^ and state dependent 

Let p,(t) = Pr {NW =" | 

exist. ' 

Putting the values of An and un in (4.3) and (4.4) we get Po айр, 
n=1,2,..., as follows: "i 

А Oey ba 
ee py p Doy Пп 39 ds (4.6) 
Pn = (u) Qu)-- qn ap 
х-л... n factors 
and p, = удуй: (ЖӨНӨ: GU). (л — 9) factors) 
№ QA)" ntes 


ssp ss 


Po 


The condition Y Р == 1 gives 


5-1 Ou) = aly)" _ *; УЫ)" ух * EAE 
mais z O 2s amo nl тйк ace di 
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5 some Ki X M 
For existence of steady state solutions, the series X (2) must 


nes 


converge, and for this to happen A/su must be less than 1. Then 


МЫ)", eF 
NE n Uum (417) 


Thus the distribution of №, when р = A/su < 1, is given by (4.6), where 
Po is-given by (4.7). | 
Note: (1) In this s-channel model, p = A/su is the traffic intensity. 
(2 When 5 = 1, we get the corresponding result for M/M/1. 
(3) The solutions p, satisfy the following recurrence relations 


1/a 
„=: (2) Pa- П = 1,2, ...,5— 1 
1/a " 
-i(i) Da n-m5,5i11l,... 


(4) The probability that an arriving unit has to wait is given by 


Pr{N>s}= $ p, 


т=з 


(Alu) ғ 
пољ pe (4.8) 


(5) Тһе expected number in the system E{N} = 5 прл can be directly 
ne 
evaluated by finding the sum of the series. Likewise E{Q}, the expected 
number inthe queue can be found (see Prabhu, Gross and Harris). 
Here we shall obtain them by making use of Little's formula. 


Waiting Time Distributions 

We assume that the queue discipline is FCFS. The distribution of Wo 
can be obtained in the same way as in М/М/І. 

Consider a test unit. It has not to wait, if it finds, on its arrival, 
less than or equal to (s — 1) units in the system. Thus 


= 
Pr {Wo = 0} = z Pa 


Au, s " 
2025 Po E (using (4.8). 


The test unit hasto wait, i.e. Wo > 0 when it finds, on arrival, that 
the number in the system is n > s, of which s are in the service channels 
and (n — s) in the queue. Thus it will have to wait for the completion 
of (n — 5 +1) Services. Asn > s, s-channels are always in Operation 
before its Service commences, and as such the time between two 
Successive service completions is €xponential with mean I/su. Thus 
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$л-з+ь being the sum of (n—s+1) iid. exponential variates with 
mean 1/ѕџ, is a gamma variate with p.d.f. 
(зш) (вих) ese 
(n — s)! 
Hence wQ(x)dx = Рг {х < Wo < x+ dx} x >0 


-i Pr {x < 5... < x -+ dx | N = n Pr {N =n} 
$ GU GUN s, 

næs (n E s)! "р, © 

. ET оо хуз т 

ог w, a(x) = в s Po 2 i 3 ee (т = My) 


= (ane Dor’ b t = 2 , (putting n — s =m) 
5: т=0 


at zi IU) ева) po. 
1; jS 
Qu Po 


© sy eye 


= Su e*s) Ps 


Rd Ju) = 
Thus- Wx) = 1— == ДЮ?” х=0 
Мет, ре“ py, >00 (4.9) 
ЕС 


and — E(Wg) = (Ио = 0):Рг {Wo = 0) [nen dxase—> 0. 


Now Јод ах = eon Рош (xe 667») dx 
f y І ТР 
n ia api e en] (Su 0) ^ sacs — р) ^v 
—— 4.10 
Thus E(Wg) = E UAE TE su(1 m p)? ( ) 


Corollary. The expected waiting time in the system is given by 
E(Ws) = E{Wo} + 1e 
Using Little’s formula, we get 


eQ Ju) 4.11) 
E(Lg) = AE(Wo) = ag Po ‹ 


а) 


(уа) 4.12 
and E(N)—AE(W3 = 5 a+ wa (4.12) 
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If the random variables, number of busy and idle channels, are denoted 
by B and J respectively, then N = О + В and I+ B = s, 
whence we get E{B} = E(N) — E(Q) = Мы 
and E{I} = s — Mu = s(1 — 9). 

Note: Suppose that instead of having homogeneous servers (viz. all 
with iid. exponential distribution with rate u) heterogeneous servers 
having independent exponential service time with rates р, і = 1,2,...,5, 
are considered. It can be seen that the waiting time density is, for 
x > 0, given by 


одо) = Уш} exp (— x CE ш 0А 
& (Gumbel). 


Example 4(a). Patients arrive at the outpatient department of a hospital 
in accordance with a Poisson process atthe mean rate of 12 per hour, 
and the distribution of time for examination by an attending physician 
isexponential with a mean of 10 minutes. What is the minimum number 
of physicians to be posted for ensuring a steady state distribution? For 
this number, find (i) the expected waiting time of a patient prior to being 
examined and (ii) the expected number of patients in the out-patient 
department. How many physicians, on an average, will remain idle? 
Here à = 12/hr., == 6/hr., so that r = A/y.—2. For steady state, p=r/s <1, 
ie. 2 < s, the minimum number of physicians must be 3. We take s = 3, 
then ¢ = 2/3. From Exercise 10.7, we find that 


E(Wg) = [sa = at PUR cub 


e~ rs 5-1 e~ rk 
where p(s, г) = mp EC et) = РА EN 


From Poisson probability taù. „ p(3, 2) = 0.180, E(3, 2) = 0.677. Ex- 
tensive tables of Poisson probability function and distribution function 
are available (Molina (1942)). : 


Ме ве: Уо) = 7; hr — 4.4 min. 


E(Ws} = E{Wo} + 1и = 13/54 hr = 14.4 min. 
E(N) = AE(W,) = 26/9 
and E(I) = s(1 — о) = 3(1 — 2/3) = 1. 


C. The Model М/М/оо 

Here the number of channels or servers are infinite so that, on arrival, 
all the units are taken into service and there is no queue. For the birth 
and death process associated with this, 44 = А, un = nu, for n = 0,1,2,. ... 
The steady state solutions can be obtained from those of M/M/s by 
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letting 5 tend to infinity. From (4.6) and (4.7) we at once get the 
Steady state probabilities, 


Ош)" 
т 


Pos, 85515 2, artsy 
where po is obtained byusing E р, = 1; we get 
п=0 


2 ОЛЫ" au 
nao nl 


ру = 
Thus puis 25 E A 1,2, anes (4.13) 


The distribution of the number N in the system is Poisson with meanA/u. 
The distribution of the number of busy channels is also the same. 


As the series У (./4)"/n! converges for all 4/u, the steady state solu- 
n=0 . 


lions exist whether or not X < и. 


D. The Model M/M/s/s : Loss system 
This model envisages that a unit, who finds, on arrival, that all the 
s-channels are busy leaves the system without waiting for service. This 
is called a (s-channel) /oss system and was first investigated by Erlang. 
This model was also examined earlier (see Examples 5(d) and 5(f), Ch. 4). 
For this birth and death process, we have 


r=), Ba = np, n=0,1,2,...,5—1 5 


= 0, =Su, nos. (4.14) 
From (4.3) and (4.4) we get the steady state probabilities 
„= шч n0, Т, кызый 
-g Ош) | Г чИ 
and li i Po=1_. or po= Pe: kl 
Thus ДЕЕ oo ЙЫ gl (4.15) 
ў We) 
k-o К! 


The probability that an arriving unit is lost to the system (which is the 
same as that all the the channels are busy) is given by 


з, TST (4.16) 


Ps s . 
> (à/u)*/k! 

k=0 

The formula (4.16) is known as Erlang’s loss formula or blocking formula 
and is denoted by B (s, Mu), while (4.15) is known as Erlang's first 
formula (or simply Erlang's formula, the corresponding distribution 
being truncated Poisson). 
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Note: (1) Attempts have been made since Erlang’s time to generalise 
Erlang’s results. Mention may be made of the works of Pollaczek, 
Palm, Kosten, Fortet, Sevast'yanov and Takacs (cf. his (1969) paper). It 
has been shown that Erlang’s formula (4.16) holds for any distribution 
of service time (having mean l/u) provided the input is Poisson (with 
parameter 2), i.e. it holds for the model М/С/5/5 (loss system). 

(See also Brumelle (1978) for a generalisation to state dependent 
arrival and service rates.) 

If N(t) is the number of busy channels at instant /, then (N(t), t > 0) 
is a Markov process, provided the service time is exponential. But the 
process is not Markov when the service time is not exponential; never- 
theless, the loss formula holds. 

(2 The steady state solutions p, exist whether or not à < p. 

(3) The results of the model M/M/co can be obtained by taking limits, 
as s— oo, of the corresponding results for M/M/s/s. 


Example 4(b). System with limited waiting time. Apart from limitation 
of space, a second kind limitation which arises in problems dealing with 
practical situations needs consideration. For example, a long distance 
telephone call may be booked to be put through within a limited time 
(say, during the office hours). If the call does not mature by that time, 
the call is cancelled and is lost. Similarly, perishable goods can often 
be stored for a limited time. In these cases, if a unit does not receive 
service, immediately on arrival, the unit waits but waits only for a limited 
time, on the expiry of which it is lost. ` 

We assume that the system has Poisson input with rate А, has s- 
channels with i.i.d. exponential service times each with mean l/u, that 
there is no limitation as regards waiting space. Further, we assume that 
the waiting time of a unit is a random variable having exponential dis- 
tribution with mean 1/v. 

The resulting process is Markov with state space (0, 1,2, . . .), and can 
be described by a birth and death process. Here àn = А for n = 0, 1, ..., 
un — np for n=0, 1, ..., s. For n> s, the transition from state n to 
state (n — 1) in an infinitesimal interval Л may occur in two mutually 
exclusive ways: either one of the s busy servers terminates his service 
(with probability и + о(Л)) or the waiting time of one of the (n — 5) 
units waiting in the queue expires (with probability (n — s) v A + О(Л)). 
Thus 

un = su + (n—s)un-scl,s42,... 
Putting the values of àn, un in (4.3), we get ` 


— Aey 
n! 


n Po ASS 


M po 
sips П {se -(r— 59 
=s+1 


„п> 5 


and Pn 


STOCHASTIC PROCESSES IN QUEUEING AND RELIABILITY 301 


and using È p,-- 1, we get 
n=O 


5 T r k nw oo Pu =i 
EE 
Pe [15 k! È us) п=з+1 ni (su (r—s)v | 


) 
г=5+1 


Е. Тһе Model with Finite Input Source 

We have assumed so far that the source or the population from which 
the input comes is infinite. We now consider the case where the source 
of input is not infinite but is finite and consists of m units. For example, 
such a situation occurs in practice in servicing of automatic machines. 
Suppose that there are m such machines, which normally do not need 
human attention except when there is a breakdown. In case of failure or 
breakdown of a machine, it is serviced by one of the s(< m) repairmen 
available. If all the s repairmen are engaged in repairing machines 
already down, the machine that is down at that time waits for repair till 
а repairman becomes free. We have a finite input source conisting of m 
machines and s (< m) repairmen. › 

Suppose that, if a machine is in working state at the instant ¢, the 
probability that it will breakdown in the infinitesimal interval (tst -+ A) 
is АЛ -+ o(h) and that, when a machine is being serviced, the probability 
that the servicing is completed in (t,: + №) is uh + o(h). Let M(t) be 
the number of machines not working (or down for repair) at the instant 
t. We have a birth and death process with parameters 


An = (т —n) А, п =0,1,2,...,m—1 


=0, nm. 
and Un = Dp», n=0,1,2,...,5—1 (4.17) 
= SU, nzs. 


. Using (4.3), we get in steady state, 


„ы qn) (т — 1))(m— 2) ...(m— n-- 1) $ 


" n!y^ o 
гы wlth an. ay 
= т = пуні ы) P 
т\ [AM 
2 m = = 4.18 
(6) Ds for 9—0, 12; ..,5—1 (4.18) 


m! 


AV г 
= use) а a 


(4.19) 
where p, is obtained by using 2 Pn = 1; we get 
ne 


ъ=[®(„)(һ) Eel]. «e 
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The probability that the number of machines in working order is r (< т) 
is given by Pm-r- А 

The model has been used as ап answer to various important questions 
of practical interest arising out of machine interference problems. Some 
such questions relate to determination of the optimal number of repair- 
men for a given set of machines, the maximum length of time during 
which all the m machines are working and so on. These have been the 
subject matter of investigation of several researchers (e.g. Palm, Naor, 
Benson and Cox; see also Feller (1968) p. 465). 

In a paper, Levhari and Shesinski (1970) used such a model in a some- 
what broader perspective: in the study of microeconomic production 
function. On the assumption that the machines produce a constant flow 
of output when operating. and that the breakdown and servicing of 
machines conformto the model of machine interference studied, they 
obtain the steady state probabilities of the system as well as the relation 
between inputs and outputand fit the same toa Cobb-Douglas production 
function. 

Remarks: n Sec. 10.4A we have considered generalised birth and 
death processes, with state dependent mean arrival and service rates A, and 
ба Оп the basis of of the results indicated there, queueing models with 
special choices of àn and un have been analysed іп subsequent subsections. 
Recently Conolly (1975), Conolly and Chan (1977) consider generalised 
birth and death queueing process, with state dependent rates A, and pn, 
as they, are. They assume the condition for the existence of steady 
state, and define a measure of traffic intensity in terms of ‘effective’ service 
time and interarrival interval. The results obtained show surprising 
similarity (with some of those of M/M/\ and with Little's formula). The 
results also indicate potentiality of application to a wider range 
of systems. 


Example 4(c). Transient solution of M/M!1/1 model. Here do = А, ш=ш 
and A, = 0, un = п, n> 1 (see eqn. (4.14)): if N(t) denotes the number 
in the system at time г, then Pr {N(t) =n} = Palt) = 0 forall n> 1, ie. 
we are concerned with only po(t) and Pit) such that pr) p,(t)= 1. 
The forward Kolmogorov equations of the model then become 


Po) = — ^p) ! upi) 
P(t) = ppt) Apt). 
Writing p,(r) — 1 — py(t) in the first equation we get Po) (р) polt) =p. 
The solution of this first order linear differential equation with constant 
Coefficients is given by 
Plt) | Се-—'%їр)! | me where C is a constant. 


Given the initial distribution P(0) - Pr (N(0) = i}, we get 


Polt) = p0) eoi. LE үр. „шу, 
ae н 
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Similarly, p,(r) = p,(0) e% + à (1 — en, 
Ad u 


The steady state solutions are 


Ро 5 lim р) = F u) 
ру = lim p(t)=N@+ ш), 
4-00 


irrespective of whether the value of p=A/u < 1 or not. 
(These agree with pa( — 0, 1) given by (4. 15).) 

Assume that the initial distribution is identical with the steady-state 
distribution so that p,(0) = Pa = B/O * 00 P0) = pi =O i р). Then 
we find that for all t > 0, и 


и 
That is, if the process is in equilibrium (steady state) initially, then it will 
be always (for айг > 0) in steady state. This is true for any other 
‘ergodic’ system. 4; 

In case of M/Mis!s . model, if the ‘system is in equilibrium initially, 
Le. Pr (N(0) = n) = pn(0) = Р» 0 << з, where p, are steady-state pro- 
babilities (given by relation (4.15), then (N(D, t > 0} becomes a stationary 
process for which M(t) has the same distribution for all t> 0, i.e. 
Palt) = Pr{N(t) = n} > Pn (given by (4.15) for all t>0 (see Takacs 


(1969)). 


A 
Balt) = Е Р» BOT рр Р 


10.5 Non-birth and Death Queueing Processes: 


Bulk Queues 4 7 
We have seen that the queueing processes in the standard models consi- 


dered in the earlier sections are birth and death processes and are, of 
course, Markovian. We examine here queueing processes arising out of 
some non-birth and death processes arising out of some non-birth and 
death processes but which are nevertheless Markovian. The Chapman- 
Kolmogorov equations can be written and the analysis can be done in a 


Similar manner. 
The basic model considere 
parameters à, p, but with group à 


MO9/M/A 


dis ofthe type of a M/M/1 queue with 
rrivals and/or batch service. 


A. Group Arrival System: К - 
Assume that the arrival streams form a Poisson process with parameter 


^ and that the number of units arriving at an arrival instant is a random 
variable X with p.f. ck = Pr (х= к k= 1, 2,.... This is аа to 
Saying that, in an infinitesimal interval (f, t + A), the probabi ity that a 
group of k arrives is Ack h | o(h). Assume here that service takes place 
singly and that the distributions of the service times are independent 

s clear that the process 18 still Markovian 


ex H 4 ean 1/ " It i ^e 
[S reque Mrs Ps process, as transitions may occur to states, not 
an - p 
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necessarily neighbouring. We denote the model by M‘*)/M/1. We get 


Do (t) = — Avot) + epit) (5.1) 
Ba (t) = — (A+ u) Palt) + UPa (t) 
+ i Аск Pat), n 2 1. (5.2) 


Let C(s) — 5 cks* be the p.g.f. of X. Since the mean arrival rate is 
kot 


^ E(X) = ^С'(1), the traffic intensity can be defined as 


о = AE(X)fg. = AC'(1)/p.. (5.3) 
Assuming that the steady state solutions exist, we get 
0 = — Apo + ups (5.4) 
0 = —(A+ u) Ps + рп +- АХ скр, п> 1. (5.5) 
Let P(s) = = Pn 5" be the p.g.f. of {pn}. 
n=0 


Multiplying (5.5) by s", for successive values of n= 1, 2,3,... and 
adding them to (5.4), we get 


0 = —A P(s) — u (P(s) — po) + (u/s) (P(s) — po) 


+A i È Ck Pn-k 5". (5.6) 


n=] kel 
Noting that the last term equals iE cy SF ( i Рак 57) = AC(s) P(s), 
= l 


we get 


ке, ы (1 — 5)p 
PO) == his (1 pm 


To find po, we use P(1)= 1. Applying l'hopital's rule, we get 


Iunio e em 
or Ро = 1 — р. (5.8) 
Thus Pis) e ЕЧ a) G =) (5.7a) 


e (1— 5) —As(1— C(s)) 
Particular case: ММ[\. This model corresponds to single arrivals, 
i.e. in groups of size one опу. Then c, = 1, c, = 0, k >1 and 
P(s) = (1 — p)/(1 — es). 
B. Bulk Service Systems 


Bulk Service Rules 


There could be a number ofa policies or rules according to which batches 
for bulk service may be formed. The following are the types of bulk 
service policies or rules frequently discussed in the literature. 
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(1) Bailey (1954) (also Downton, 1955) considers that units are 
served in batches of not more than b (say). If,immediately after the com- 
pletion of a service, the server finds more than 5 units waiting, he takes a 
batch of b for service while others wait; if he finds r units (0 < r « b), he 
takes all the r units in a batch for service. Bloemena (1960), Jaiswal(1960) 
Neuts (1967) consider the same rule with the restriction that г == 0 
(1 << г < D), i.c. the service facility stops until a unit' arrives. This rule 
will be called the usual bulk service rule, while Bailey's rule will be its 
modified type (it will be also called bulk service rule with intermittently 
available server). Jaiswal points out that the distribution of the queue 
length for the modified rule can be obtained from that of the usual rule. 

(2) The rule with a fixed batch size k has been considered by Fabens 
(1961) Takacs(1962) and others. In this case the server waits until 
there are k units. 

(3) Neuts (1967) considers this rule: If, immediately after the com- 
pletion of a service, the server finds less than a units present, he waits 
until there are a units, whereupon he takes the batch of a units for service; 
if he finds a or more but at most b, he takes them all in the batch and if 
he finds more than b, he takes in the batch for service b units, while 
others wait. The batch takes a minimum of a units and a maximum of 
b units. This rule will be called general bulk service rile as rules under 
(1) and (2) can be taken as particular cases. 

(4) Bhat (1963) considers the rule that the number taken in a batch 
is a random variable Y. The corresponding Markov model is denoted 
by M/M@)/1, У 

There are actual situations where one гше seems to be better suited or 
to be more appropriate than the others, Examples are given in standard 
textbooks and relevant research publications. 


Poisson Queue with General Bulk Service Rule: M/M?^/1* 

Assume that the arrivals occursingly in accordance with a Poisson 
process with parameter А. The units are served in batches according to 
а general bulk service rule such that the service starts only when а min- 
imum number of units, say, a is present in the queue, the maximum 
Service capacity being, say, b. The service times of batches of size s 
(a< s < b) are assumed to have independent identical exponential distri- 
bution with mean l/p. For simplicity, the service time distribution is 
assumed to be independent of the batch size s. Denote by state (0, q), 
the state that the service channel is idle and there are q(0 «qx a—D) 
units waiting in the queue and by the state(l, п), the state that the 
DONE channel is busy and there are n (л > 0) units waiting in the queue. 

et 

Po,q(t) = Pr {that at time ż, the system is in the state (0, q)1, 
@= 0; 1,25 asi 


*The queueing system is denoted here by M/M^*/1. 


20(45-95/1982) 
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Pa,n(t) = Pr {that at time г, the system is in the state (1, п)}, 
nii ni 2, га. 
Assume that the steady state distribution exists and let 


lim ро,а(#) = po, 
too 
lim py,n(t) = Pra- 
too 


The steady state probability equations are: 
O= — (à + и.) Pus + AP ina Pise п = 1,2,..., (5.9) 


b 
= — (A+ и) Pao + Аро,а—1 + к È Pus (5.10) 
0= — Doo + E Pro (5.11) 
O= — Mon + ona + UPum п 1,2,..., а. (5.12) 


Note that (5.12) will not occur when а == 1. First we solve (5.9). 
The difference equation (5.9) can be written as 


WE) {Pi 0), 25930, 1,2. (5.13) 
such that the characteristic equation is * 
h(z) = pz! — (A + p) z A = 0. (5.14) 
Suppose that (2) = – (0 + и) 2 and that g(z) = uz5" +, 


then for |z|=1, | g(2) | < |/@) | and hence by Rouché's theorem, 
Ҳа) and f(z) | g(z) will have the same number of zeros inside |z|21. 
Since f(z) has only one zero inside | z | = 1, there will be only one zero 
of f(z) + g(z) = h(z) inside |z | = 1. 

Denote this root of A(z) = 0 by r, 0 <r < 1, and the other b roots by 
Fas esos Fibs (| | ЭЭ 1). 

Denote the traffic intensity by р = A/bu. From (5.14), we get. 


bp =d t=”) 


z EF =r +r +... Hrb. (5.15) 


Now we can write the solution of (5.9) as: 
Pin = Ar" +E Ат", n=0, 1, 2,... 
Since 3 Pum < 1, we must have 4; = 0 for i= 1, 2, ..., b,so that, 
j Pirn = Ат" 
= (Pi,o)r" 


= ОЛ)" ро, o. (from (5.11)) 


uo mu З 
=( 1 er Do, (5.16) 
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Using (5.16), we get from (5.10) 


1—г 


Po, а-1 = рр Po (5.17) 
and finally putting л = 1, 2, ... a— lin (5.12), afd using (5.16) and 
(5.17) we get, recursively for q = a — 2, a — 3, ..., 2, 1, 

]— r 
Pyme ЧЛ (5.18) 


and because of (5.17), (5.18) holds for q = 1, 2, ..., a— 1. 
a-l со 
Since Z py, Z рп = l, we have 
9-0 n=0 


a pati peni -1 
Poso = [к= | s (5.19) 


Thus (5.16), (5.18), (5.19) give the steady state probabilities. 
Particular cases; Fixed size bulk service : M/M***/1. 


Неге а =b = k. Letp, denote the steady state probability that the 
number in the system isn. Then 


Po = Розо = Gat 


1 à 
Pa = роза == eme eel - rk, q = 1,2, ра EESTI S 


Pingk = Piym = (Ми) por" = e(1 — r) r", e = i. ae 
mss, 179... 


ie je fe i= 0, ое 


e(l—r)r-*,n=k, kt... ab 


Distribution of the waiting time for the system M/M*/1 (Medhi, 1975) 
Assume that the system is in steady state. Let the random variable T 


denote the waiting time in the queue foran arriving unit. 
Let 


v(t) be the p.d.f. of T, 
S (a, kit) be the p.d.f. of gamma distribution with parameters œ, К, 
Lè; 


J К) = ок econIT(K), t > 0, k=- 1,2, ..., 


Г, (о, К) Бе the incomplete gamma function, i.e. 
x 
Té) = [re ina 
LU 


k-i 
—l1-— Ў e-**(axyjs!. 
s=0 
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An arriving unit may find the system in one of the following states: 

© (0,а—1) 

G) (0,4), 0<4<а—2 

(ii) (1,7,a—le&m«b—l) n=kb+m 

(iv) (12,0 m«&a—2 = 001,9... 

In case of (i), the arriving unit does not wait; the probability of zero 
delay is thus ро,а-1. Inall other cases the unit has to wait, the probab- 
ility of blocking (or delay) is thus 1 — poa ,. 

In case of (ii), the arriving unit has to wait the arrival of (a — 1—4) 
units and the time needed for (a — 1 — q) arrivals has a gamma distri- 
bution with parameters A, a — 1 — q. 

In case of (iii) the arriving unit has to wait for the completion of 
services of (k+ 1) batches; the time required for this has a gamma 
distribution with parameters p, k+ 1. 

In case of (iv), the arriving unit has to wait till either the services of 
(К + 1) groups are completed or (а — 1 — m) units atrive, whichever 
occurs later. This duration whichis given by the maximum of two 
gamma variates may be denoted by the random variable Z, i.e. 


(5.21) 


2 = max {gamma variate with parameters A,a— 1 — m; 
gamma variate with parameters p, k + 1). (5.22) 


The distribution function Fz(t) and the p.d.f. hz(t) of Z are given by 
Fz(t)= Pr {Z< t) = Г, A, a— 1 — m)T, (и, k+ 1), 
and hz(t) = F'z(t) 
=f, a— 1 — m;t) Г, (и, k+ 1) 
+T: A,a— 1— m) f (y k +1; t). (5.23) 
It follows that the p.d.f. v(t) of Tis given by 


a-2 
у@) = c Poa fA, a —1 — q; t) 


со b-i 
FD „2 Рокн f(e k+ 1; 1). (5.24) 
со а—2 
t „2 Рот hz(t), 0 <г< о. 


It may be verified that 
[ v(t) dt=1— Po, а-л» 
0 
as is to be expected. 
Substituting the values of ро, г, ру; and the expressions for Sand h in 


(5.24), we get an exact expression for the p.d f. v (t) as follows: 
Denote 


m-i 
= zkjk! = e(m, 2); e=. e(m, z) = E(m, z),m>1. (5.25) 
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Then 


= га, a—q— 1; t) =Art-? exp (— At) e(a — 1, At[r). 

q=0 

Thus the first member on the right hand side (r.h.s) of (5.24) becomes 
Apo, [E (a 1, t) — rt! exp (—M)e(a—lat/AV(l—r). (5.26) 


The second member on the r.h.s. of (5.24) is equal to 
5-1 E 
(ш) Poo [{ Z р т") {u exp (— ut) „4 (e #7) 
m-a- {е 


=[Аро (rt — 15) exp { — e (1— PNA т. (5.27) 


The third member on the r.h.s. of (5.24) consists of two terms correspon- 
ding to those of (5.23); the first of these becomes 


(lt) Poso [f= г (Ма—1—т; oH | à (r9 f (и, k+1; х) ash] 
0 


= Apo, [777 exp ( — At) e(a — 1, at/r)] 
"П — exp (— u(1 — 79 HY — r), (5.28) 


and the second becomes 


Олдрь1® CHF fps Е+1; mez Por 1-m 


Tuc a-g-m е-№ (М) 
= Аро,о[ехр {— (1 — 79) 0i [ zn wes E een) 
— r E(a-lAt 
sa lene ma (1 — 1 jA- 26200 
re-l exp ( — M) e(a — 1, Mjr) 29 
Шз е ЕРСИ" xm E I G. ) 


Hence adding the expressions of the r.h.s of (5.26) to (5.29), we get 
v (t) = Das — IQ — 2) exp {— p (1 > 7980 
+ E(a — 1,А0) {1 — exp ( — (1 — 05)0)], 0 < t < о. (5.30) 


the conditional waiting time distribution, given that a 
o wait (i.e. an arriving unit who finds the service channel 
han a — 1 waiting) is given by 


w (t) = V (DHL — Posi 


The p.d.f. of 
customer has t 


n i ith less € 
busy or idle wi 65.31) 


зр Л 
ке бз SU А note that for positive integral values of 
nd tho , 
k and for a> 2, 
es 
х= | exp {— el 
0 


k+1 
1=9):)4 = а, (5.32) 
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1,0)2 | *£«— 1,504 
о 


| 
Ш 


zf tkt) e™ dtjs! 
8-0. A 


"3" в 16 4 2)...G рк, (5.33) 


5=0 


1 


Ják) = | tk E(a — 1, At) exp (— u(1 — rb) t} dt 
o 


= “® (s+ Ds +2). . (s + K) reiten, (5.34) 
s=0 . 


The expected waiting time in the queue E(T) is given by 


щту= | jar = Mes ta — у л) 40) — л) | 


0 
— Moo Г 1 a(a—1) | art (1 — 5) — n — =| 
“Trl а tr [| 


(5.35) 
It can be verified that Litt!e's formula holds. 
Higher moments can be obtained in the same manner. 


Particular case: M/M**/1, For the usual bulk service rule, 
E (1— ғ) (=? 
I=rF Oe) ASES 


Po-0 


(09 (100) 
Pin = Tr + rad т) 
The waiting time density is given by 


pU m0, E x 


V(t) = Dpo,o/(t — (I — 75) exp (7 &(1 — 75) 19], (5.36) 
and the expected waiting time by 


Р 
FO) = тект) Ge? 
The conditional waiting time distribution of only those who wait is 
exponential with mean 1/u(1— r^). 
Putting 5 —1 (when r—9) we get the corresponding results for 
мім]. і 
Remarks: Subsequent to Neuts (1967), who introduced the general 
bulk service rule for single channel systems, such models have been 
studied by Teghem et al. (1969), Medhi et al. (1972), Medhi (1975, 1979), 
Borthakur (1975, 1976), Neuts (1979), Sim, Templeton et al.; Arora 
(1964), Ghare (1968), Cromie et al. (1976), Chaudhry et al., Weiss (1979), 
and others examine some similar bulk service models, 
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10.6 The Queueing Models М/С/1 and GI/M/1 


So long we have been discussing queueing processes which are either 
birth and death or non-birth and death processes. They are in either 


case Markovian and the theory of Markov chains and processes could 
be applied in their studies. We shall now consider models where the 


distributions of the interarrival time or the service time do not possess 
the memoryless property, i.e. are not exponential. The process N(i) 
giving the state of the system or system size at time і will then be no 
longer Markovian; however, the analysis of the process can be based 
on an associated process which is Markovian. Two techniques are 
generally being used for this purpose. Kendall (1951) uses the concept 
of regeneration point (due to Palm) by suitable choice of regeneration 
points and extracts, from the process (N(t)). Markov chains in discrete 
time at those points. This is known as the technique of imbedded 
Markov chains. The second important technique (due to Keilson and 
Kooharian (1960)) known as supplementary variable technique, involves 
inclusion of such variable(s). 
We discuss below Kendall's method. 


A. Queues with Poisson Input: model M/G/1 
Assume that the input process is Poisson with intensity à and that the 

service times are iid. random variables having an arbitrary (general) 
distribution with mean 1/и. Denote the service time by v, its d.f., by 
B(t), its p.d.f., when it exists, by b(¢)( = В'(1)), and its L.T. by 

BY(s) = [ e= dB(t). 

0 
Let t&,n—1,2,..., (to = 0) be the nth departure epoch, i.e. the 
instant at which the nth unit completes his service and leaves the system. 
These points ¢, are the regeneration points of the process {Ni?)}- 
The sequence of points {fn} forms a renewal process. N(t,-+ 0), the 
number in the system immediately after the mth departure has a 
denumerable state space (0, 1,2, ...). Write (і, + 0) = X, n-90, 
l, ... and denote by An, the random variable giving the number of 
units that arrive ducing.the service time of the nth unit. Then 
Xa = Xn =* 1+ Anas if X, z 1 
= Ansar if X, = 0. (6.1) 


Now the service times of all the units have the same distribution so that 
A, = Afor n=1,2,.... We have 


ч n EFA е-М (м) 
Pr {4 = г | service time of a unit is j= е О 


r! 


andso k = Pr(A—r) jae dBt),r=0,1,2,..4, (6.2) 
o 
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gives the distribution of 4, the number of arrivals during the service time 
ofa unit. The probabilities 


Pij = Pr {Xnr =I | Xn = i} - 
are given by я 4 
Фир == Кула» i> 1,) 201—1 


=0 ‚ір 1,ј<1-1 (6.3) 
Poj = ру = К, j > 0. 


The relations (6.3) clearly indicate that (Y, n > 0} isa Markov chain 
having t.p.m. 


ko А АҺ 
u AG 5 
=(р)=|0 k č k ..| (6.4) 


As every state can be reached from every other state, the Markov chain 


{X,} is irreducible. Again as p; 2 0, the chain is aperiodic. It can ` 


also be shown that, when the traffic intensity р = А/ш < 1, the chain is 
persistent, non-null and hence ergodic. We can then apply the ergodic 
theorem of Markov chain (Theorem 3.6). 

The limiting probabilities 


= lim pf9, j =0,1,2,... 
noo 
exist and are independent of the initial state i. The probabilities 
V -= (voti...) È vy = 1, are given as the unique solutions of 
V= VP. 


Let K(s) -Zkjs and V(s) = E vs) denote the p.g.f. of the distribu- 
tions of {kj} aud (vj) respectively. 


We have K(s) — X Куз) = Efe 700 аву) 
Ј=0 
ô 


ос 


= Sw dB(t) 


= В* (у. — As). ‚ 6.5) 
Hence E(A) = K' (1) = — АВ*0) (0) = A/a = p. (6.6) 
Now V — VP gives an infinite system of A 
Multiplying the (k + 1)st equation by s*, k = 0, 1l,... and adding over 
k, we get, on simplification, for 0 — e«l, 
== К'(1)} (1 — 5) K(s) 
Voy K(s)— s 


see Example 5(d), Ch. 3 
Putting K' (1) =p, we get ( P ( » ) 
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И) = OL К) (6.7) 
_(1—@(1—) BAAS) 
=т= ДЕСЕ =E Я (6.7а). 


This is known as Pollaczek-Khinchine (Р.К.) formula. 

Remarks: For the mode! М/С/1, the process (N(0)) giving the system 
size at time ¢ is not a Markov process. By extracting з process at 
departure points £,, we get an associated process X; = М, + 0), which 
isa Markov chain. The transition probabilities, pj) of the Markov chain, 
give departure point probabilities, while 


pi (f) = Pr£NQ) =n | NO) = д 


give the general time probabilities of the system size. When the 
Markov chain is ergodic (the condition for which is p < 1), the 
limiting probabilities v, — lim pin), asm — œ, exist and can be obtained 
by applying the ergodic theorem of Markov chain. When the steady 
state exists (the condition for which is also p < 1) the limiting probabi- 
lities p, = lim pin (t), as £ — со exist. It can be shown that (for proof, see 
Cohen, Gross and Harris) when р < l,then Vn = ps for all n, ie. the 
limiting departure point probabilities are equal to the corresponding ` 
steady state probabilities of the system size. Thus ergodicity of the 
Markov chain is equivalent to the existence of steady state, the condition 
for each to hold good being р < 1. 

It may also be noted that another interesting stochastic process can 
be extracted. The process Y(t) = Xs, tn S É < tata gives the number 
of units left behind in the system by the most recent departure. Clearly, 
{Xm tn} is a Markov renewal process; and (Y(t), t2 0) isa semi-Markov 
process, X, being its embedded Markov chain and the waiting time in а 
state being distributed as the service time v. We shall consider such 
semi-Markov processes later. 


Example 6(a). The Model M/E;/1- Consider that the service time v is 
Erlang-k (with mean 1/0) having p.d.f. 


$e _ (pkk tk- ev kur 
ави) = Би) = Ry 0 =. 


" Es ule NE : 
Its L.T. is В* (5) = (a Т т) (see Example 3(f), Ch. 1). 
From (6,6) and (6.7) we get 


k 
К) = В" a-n) = (к=п вк =x) 


= (14 202)" оме 
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(i — 9) — 5) 
and MI = Sea (Pap 
Expanding in powers of s, and comparing coefficients of 5" from both 
sides, we get v,, and equivalently p,. We can find the moments of {Pn} 
from V(s). " 

This model could also be considered as non-birth and death Markovian; 
by considering that the service comprises of k exponential phases each 
with mean 1/(ky), the C.K. equations can be written and solutions 
obtained therefrom. 

Special case: The Model МІМП. Putting k = 1, we at once get 
(for M/M/1), ü xt À » к 

a — 011 — 5 as! =e 
KG) О (l—s—sp(1—s)) (1—зо)” 


Example 6(b). The model M/D/\. Аз К — co, the whole mass of the 
distribution Ex tends to concentrate at the point 1/p, і.е. Ey =D, as 
k — oo. We have then 


B* (s) > est, 
so that 


K(s) = B* (у, — As) > e709) 
and 


= U9) =з) 
o= Ga ш 
Expanding the r.h.s. in powers of s, and after some algebraic manipula- 
tions, we get v„, the coefficient of s", n = 0,1, 2,..., as follows: 
Yo = 1 — р 
у= (1 — eer — 1) 


= = п-1 а T (rey'-r rp)n-r-i 
y, = (1 — 8) [e 2 (— 1)'-r ere lee arm n22: 
WAITING Time: POLLACZEK-KHINCHINE FORMULA 
Assume that the steady state exists (p — 1). 
Let W and Ио denote the waiting times in the System and in the queue 
respectively and W(t) and Wo(t) denote their distribution functions. 
We have 


Pn = Vn = Pr {departing unit leaves п in the system} 
= f Pr {departing unit leaves n in the System | t < W << t + di) 
0 
Ж РС W< tt dt} 


= |007 amo n =0,1,2,.... (6:8) 
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Thus 
co 
V(s)= È ws" 
n=0 


= [te ano] $ Our ooh 
axo n 


о 


E | етм 0-3 dW(t) (6.9) 
0 
+ W* (у — А5), 


where  W#(0) = | e" dW(t). 
0 
Now, P(s) = È pas" = Z vus" = V(s) 
= (А —As). (6.9а) 
From (6.7), (6.9) and (6.6) we get 
W*( — às) = йе D) ecl ыыы ын M 
Writing « for à — As, we get 


| (d = et (а/л) B*() 
W* (9) = Bie) — (1 — ey) 


_ «(1—9 В*(а) 
^ .— XQ -— В") (6.10) 
Or, м 

s(1—9 В%(5) . (6.102) 


Ws) B*3) 


This relation connects the L.T.'s of the distributions of the service time 
v and the waiting time W in the system. 
Now, W = Wo + v, so that W*(s) = Wo*(s) B*(s), Wo” being the L.T. 
of Wo. Thus, we have 
qe. baee 6.11 
Wo*(s) = р B*()Y (6.11) 
The relation (6.102) (as well as (6.11)) is also known as Pollaczek- 
Khinchine (P.K.) formula. To distinguish it from the other P.K. 
formula (6.7), it is called Р.К. transform formula. The formula has been 
Tediscovered in special contexts by different researchers. There are many 


interesting direct proofs of the formula (see also Runnenberg (1960)). 


Expected Waiting Time and Expected Number in the System 
Using (6.7) and (6.11), moments of E(N) and Е{ Wo} can be obtained. 
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к 
We have B*(*)(0) = B*(s) = (— 1) Ev), k= 110 К (6.12) 
S-20 
From (6.11), we get 
Ws) _ s —a + ABS) — 5{1 + AB*O(sy 
паа {s — AE AB*(S)H > 
sich takes the form 2 as s— 0. 


We can use L’Hopital’s rule to obtain its value as 5 0. 
AS s — 0, the r.h.s. becomes 
Q + 2B*(s)) — (1 + AB* (sj) — S(AB*GXs) — — AB*OX(0) 
2s — А -- AB*(s)) (1 + ABN) ATF ABOO) 
" AE(v?) 
201 


a 
- *() e 2 
Thus, E(Wg) = Wg*e(0) (1—6) Eç’) 
E аг. A ) 6.13 
xi-3(* tpe om 
where o,? is the variance of the service time v. 


By differentiating Wo*(s), k times w.r.t. s and putting s — 0, we can 
obtain the kth moment of Wo. It can be seen that 


А А О, 
i= — p) E(p3) + Зу] (6.14 
var Wa} = top [4 (1 — р) E0) + ЗА (Е бе) (6.14) 
It can also be shown that (Sphicas e; al., 1978) 
c > (4 — 9/30 > 1, 

where c — s.d. {Wo ELW o} is the Coefficient of variation of Wo. 
Note: By utilising a relationship between Single server queues and 
random walks and without using P.K. formula, Lemoine (1976) obtains 


the moments of waiting time of M/G/1 queue. 
The mean waiting time in the system is 


ELW} = E(Wo + v} = Ev) + ELWa) 


o AC | 
ut Xi-g £05. (6.15) 
Using Little's formula, 


we can obtain E(N) from (6.15) and E(Q) from 
(6.13). We bave 


E(N) = AEQV) = р "3-3 Еф) (6.16) 
and F(Q) = ЕШ} = р. Боз 


E{N}, E(N?), ... can be Obtained directi 


У from (6.7). The formula 
(6.16) is also sometimes called P.K. fo 


rmule. Following Kleinrock 
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(1976) we may callit P.K. mean value formula, to distinguish it from P.K. 
transform formula. 

From (6.13) and (6.15) it becomes at once clear that for M/G/1 queue 
with parameters à, p, expected waiting time, whether in the queue orin the 
system, is the least when c, = 0, i.e. service time distribution is constant 
(= ш). The model then becomes M/D/l1. Thus, for all single server 
models with Poisson input, the one with constant service time has the 
least expected waiting time and hence the least expected queue and system 
size. 

We discussed the busy period for the M/M/1 model in Sec 10.3; here 
we consider it for M/G/1 model through an alternative approach. 


Busv PERIOD 
The expected duration of the busy period 7 (initiated by a single custo- 
mer) follows immediately from the result noted in Example 8(a), Ch. 6. 


For p < 1, we have 
а Bis oe et E(D) 
Por dit WOU oye RT) 


Clearly, the idle period / here is exponential with mean 1/2, since the 
interarrival distribution is so. As stated in the remarks noted earlier in 
this section, p, = V» for all. From equation (6.7) we find that v, the 
constant term in V(s) is given by v, = V(0) = 1 — р, so that 


кы); 
Pom 1 — = TRE ЕТ) 


1 Ev) 
whence. E(T)= Ban bep 
Thus, given the mean arrival and service rates, the expected duration of a 
busy period in a queue with Poisson input is independent of the form of 
the distribution of the service time. 

We derive below an integral equation satisfied by the d.f. of а busy 


period, initiated by a single unit or customer. 


(v being the service time). 


Takacs Integral Equation 
Let v be the service time, 


B(t) = Pr(v < t), B*(s) = | e-" dB(t), 
0 


and T be the duration of the busy period, 
G(t) = Pr {T< t}, G(s) = | e=" dG(t). 
0 
Let G"* denote the n-fold convolution of G with itself. Let A(x) denote 
the number of arrivals in [0, x]. 
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By conditioning on the duration of the service time v of the initial 
customer inaugurating the busy period and on the number А of arrivals 
during his service time, one gets, 

t 
Git) — е B(t) 4 2 | Pr {T< t | v = x, A(x) ^ n) dB(x) Pr{A(x) -- n); 


n=1 


0 


the first term corresponds to the case that none arrives during the service 
period of the initial customer (and then x =- t). Now let 7; be the total 
service time of the ith descendant} and of all his progeny, Then T; are 
i.i.d. and have the same distribution as Т. 


We have 
Pr {T <t | v =x, A(x) = n) = Pr{T,4+...4- Ta t—x) 
exe (1—9), л>. 
Writing G’* Т, form = 0, and noting that then x = f, we get, 


t 
* к ex) 
G ( ) 2 n! 


С"* (t — x) dB(x). 
This equation is known as Takacs integral equation. 
Putting the above expression for G(t) on the r.h.s. of 
G*(s) =| e dG(t), 
0 


changing the order of integration in the repeated integral (on the r.h.s.) 
and on simplification, we get 


G*(s) = B*(s +- X — à G*(s)), 


which is a functional equation in terms of L. T. In view of its import- 
ance, we give an alternative derivation of the above equation. 

Given that v = x, i.e. the initial customer (“ancestor”) inaugurating 
the busy period departs at x, the number of (‘descendants” ie.) 
persons N arriving by time x has Poisson distribution with mean Ах. 
Thus given v = x, the duration of the busy period T equals x + Sy, 
where 


Since the L.T. of T is G*(s), the L.T. of Sy is 


exp(— Xx -- Ax G*(s)} (by Theorem 1.11) 
Thus the L.T. of the r.v. T, given у = x, is 


TIn branching process terminology, the initial customer may be called “ancestor”, 
those who arrive during his service time may be called (direct) “descendant” and the 
descendants of the descendants and so on, may be called **progeny" 
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Е{е-#Т | у х} = exp ( — ѕх) exp { — Ax |- Ax G*(s)} 
= exp { — x (s -HA — à G*(s))). 


Finally, 
G*(5) = ET. of T= Exe} 


= f E(e^T | v = x) dB(x) 
о 


= | exp (— x(s + A — à G*(s))) dB(x) 
0 


== B*(s +a — ^ G*(s)). 


As can be imagined, such an equation would occur in branching processes. 
The reduction of the busy period to branching processes is due first to 
Good, whose heuristic arguments leading to the equation were given 
formal content by Takacs. We state the following proposition: , 

Let H(s) be the L.T. of a distribution with mean b and let à > 0, then 
the equation 


K(s) = H(s-- 4 —2 K(s)) 
has a unique root K(s) < 1; further, K(s) is the L.T. of a proper pro- 
bability distribution iff Ab « 1. 

(For a proof of the above proposition, see Feller, Vol. II.) 

From the above proposition it is clear that G*(s) is the L.T. of a 
proper probability distribution if e -- х/ы < 1. If ọ > 1, there is a non- 
zero probability of a never-ending busy period, the probability being 
equal to the smallest positive root of the equation ¢ = B*(A — XC). 

Takacs obtains the inversion of G*(s) in the form 


G(t) = Cu oy dB (t). 
n=1 
0 


Even without finding G(t), one can find the moments of T from the 
functional equation in G*. As can be easily verified 


ET) = 0) 


am. _ (EO 
=) (= 

As a special case, we can obtain. G*(s) for M/M/1 model from the 
functional equation by putting B*(s)=- u/(s-- p). The equation then 
reduces to a quadratic in G*(s), whose root is seen to be 6/p (as given in 
equation (3 .23)). 


and var (T) = 
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Number Served During a Busy Period 

Denote by N(T), the number of customers served during a busy period 
initiated by a single customer (‘‘ancestor”) and by Р(2), its p.g.f. Denote 
by МТ), the number served during the total service time 7; of the jth 
direct descendant and of all his progeny (i.e. N(7j) equals 1 plus the 
number of progeny of the jth direct descendant). Clearly 


МТ) 1--® NT) 


(A(T) gives the total number of persons ever born in the family includ- 
ing the ancestor before the family becomes extinct). 

Given that the initial customer departs at x. the number A(x) of arrivals 
(direct **descendants") is a Poisson variate with mean Ax. Denoting 
P(z) -- 5, we get 

E(zNT) | у x) есл, 


Hence 
E(zNOD | y = x} == тех), 
Thus we get 
Ee гт) | E(eN@) |v +x} dB(x) 
0 
=z je dB(x), 
0 
ie. Е = zB*(. — 28). 


Thus 5 satisfies the above functional equation. It can be shown that the 
equation has a unique root £(z) such that &(0) == 0. 

A fundamental result concerning the joint distribution of the busy 
period T and the number N served therein for a M/G/1 queue has been 
given by Prabhu (1960, 1965); (see Enns (1969), Scott et al. (1972) for 
related trivariate distributions). 


The model M|G][co: transient solution 

Suppose that the input is Poisson with intensity à, the service time v is 
general with distribution function B having mean 1/y and that there is an 
infinite number of channels so that an arriving unit is immediately taken 
into service whatever be the number present in the system. Let N(t) 
denote the number present in the system (or number of busy channels) 
at epoch t, and p,(t) == Pr (N(t) = n}. Assume that initially the system 
is empty, i.c. po(0) = 1, pa(0) —0, n 40. If X(t) denotes the number of 
arrivals by time г, then Pr (X(t) = k} = e-™ Quf)'/k!. If Y(t) denotes the 
number of departures by time t, then X(t)=N(t) | Y(t). By conditioning 
on the number of arrivals X(t), we get 


Palt) = Pr {N(t) п} B Pr (N() = n | Ха) = К}-Рг (X(t) = К). 
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Let a = a(t) = Pr fan arbitrary unit arriving during [0, 1] is still in service 
at epoch г). Then, conditioning on the epoch of arrival, we get 

t 

a =Í Pr {service time of the arriving unit exceeds г — x | arrival 

à . occurs at epoch х}: Pr (arrival occurs at x} dx. 
Now, arrival is in accordance with a Poisson process and so, given that an 
arrival occurs in [0, г], the distribution of the arrival time is uniform in 
[0, t] (Theorem 4.5). Thus, 

t 


=| [Pr (v — x} (1/0] dx 


0 
t 


=(i/t) fu — B (t — 3) dx = (00) fa — B(u)) du 
0 0 


t 


and 1-58 = (1) | Blu) du. 


à 
Now, Pr(N(t) = т | Xt) -- К} = Pr {given that k arrived in 0, /], the 
number in service at epoch / is л} 
(5 Jea 2 кп. 
Thus ' 
um [EX ла Aen OO 
0 2 (, Ja @ — а= ex OP 


п! CER 


ы [= ем “фе” |$ P Ке 2 nd 
ken (k = n)! 


hat п 
ECCU ESQ coo 


n! 


t t 
= exp {- A | {1 — B(0)) UE ja — Bu) du fu } 
0 0 
That is, the distribution of №?) is Poisson with mean 
1 


hat =- А | (1 — В(и)} du. 
0 
Similarly, it can be shown that the distribution of the departure process 
Y(t) is also Poisson with mean 


1 
a(l = а)? = А | B(u) du. 
9 


21(45-95/1982) 
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Particular case: (1) Steady state solution. 
t 
As t — ©, a-[a — Bou} du = 1, 
0 
so that, Aat — Аи =p. Then the steady-state distribution of the number 
in the system in М/С/оо modelis Poisson with mean р = A/p, irrespective 
of the value of p. 
(2) M/M]/co in transient state with p,(0) = 1. Неге В (1) = 1 — e- 
so that Aat = (А/ш) (1 — е8}. Thus, for n € N 


моа eem 


As t— ©, р (1) ә e? р"/п!, ne М. 

Thus, the steady-state distribution of the number in the system in a 
model with Poisson input is, irrespective of the form of the distribution 
of the service time as well as of the value of р, always Poisson in case of 
infinite channel model. As mentioned in Note (1) to Sec. 10.4 D, the 
distribution is always truncated Poisson in case of a s-channel (105%) 
model. 


An Application in Reliability: A spares inventory model (Barlow and 
Proschan, 1975). 

Consider a maintenance depot with a large number of repairmen. 
Suppose that the failure of units occur in accordance with a Poisson 
process with rate 4 Failed units are brought to the depot for repair, the 
repair time v being i i.d. random variables having common distribution 
function B(r) = Priv < 1) and mean l/u. Suppose that repair times 
are also independent of the arrival times and that the repaired units 
become part of the spares inventory, which consists initially (at г = 0), of 
a certain number of spares. Let M(t) denote the number of repairmen 
engaged at repair at epoch г and let N(0) = 0. 

Then we have an М/С/оо model for the spares inventory and the distri- 


Li 
bution of N(t) is Poisson with mean à [ {1 — B(u)} du. 
0 


If п is the number of spares in the inventory at £ = 0, then the expected 
number of customers waiting (or spares at epoch r is given by 


; E [max (Аи) — n), 0j] 
= 23 (k—n)-P =k 
A n)-Pr {N(t) = k}. 
As t — œ, this expected number tends to 


a n) k! C 


In» 
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B. The Model GI/M/1 

Here we assume that the service time distribution is exponential with 
mean l/u and that the interarrival time is a random variable и, having 
an arbitrary (general) distribution with mean 1/4. Denote the d.f. of 
u by A(t) and its p.d.f., when it exists, by a(t); its L.T. is given by 


A*(5) = [ e dA(t) (6.17) 
| i 
dk 
юм Ано (= AP) |= (— NF EO) (6 18) 
АУ 


and for k= 1, A*@) (0) = -= 1/х. 

Let tn n=1,2,..., (= 0) Бе ће epoch at which the nth arrival 
occurs (or nth unit arrives). The process N(t, — 0) = Yn n = 0, 1, 2,... 
gives the number in the system immediately before the arrival of the nth 
unit. Then 

Кы = Yn + 1— Ва, if Y, > 0, Ва < Улы, 
where B,4, is the number of units served during (i4 — tn), i.e. the 
interarrival time between the nth and (n + 1)з unit. As В, is indepen- 
dent of n, i.e. B, = B for all n, its distribution is given by 
© -ut 
g = Pr (B= r) = (EEY AO, r=0,1,2 (6.19) 
0 
The arrival point conditional probabilities 
pij; = Pr (Yn, =j| Yn =i} 
are given by 


Pij = &на— Ù F L>j2 ht> 0 (6.20) 

=0, i+1l<j,* ; 

141 
апі У р = 1. 

Ј=0 0. 

а idi 
=1— È 2 

Thus Pio =1— A ру = hd Bin- 


eT i gr = М (say), 2 0. (6.21) 


r=0 , А 
і hain having 
Since all p;, deperd only on i and J, (y, nz0jisa Markov chai 
t.p.m. a A Oe 
hy Lco. МАК ү (6.22) 
Р=()=| i a ® b 


p i те + dic. It сап also be shown 
in is i ucible and aperiodic. 
aa ap ep AT pee non-null when р < 1. Thus, when e « 1, 


that it is ergodic, ре 
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i.e. in the ergodic case, the limiting arrival point system size probabilities 
v; = lim 00 
n> 
exist and are given as the unique solution of the system of equations 
VSVP; (6.23) 

where V= (Noy. xs), ре П, 
We now proceed to find V. 

Let G(s) - X 8, s" be the P.g.f. of {g,}, i.e. of ће r.v. B, the number 

r=0 


of units served during an interarrival interval. We have 


G(s) = È g,s" 
r=0 
= | HOO say 
fel rt 
о 


== | e-vO- 9! A(t) 


0 


= A*(u(1 -- 5)). 
Also E(B) = G'(1) = — pA*) (0) , 
== usr = le. 
The equations (6.23) сап be written as 
n= È wh (6.24) 
r=0 
and y= 2. шн 8» j21. (6.25) 


Denoting the displacement operator by E (so that Er (vj) = ур. etc.), we 
can write (6.25) as a difference equation 


Еу) =v) = ži Sr Еу), ] > 1 


ог, {Е— È g E) 9.) =0 


or, {E — G(E)} vj.,—0,j— 120. (6.26) 


This equation can be solved by the method given in Sec. 1.5. The 
characteristic equation of the difference equation is 


(0) == — G(z) =z — A*(y — uz) = 0. 


It can be shown that when G'(I) = 1/ẹ > 1, ie. p < 1, r(z) has only one 
zero inside | = | = 1. Assume that p < 1, then, if the root of r(z)=0 
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inside |z| = 1 is denoted by r, and the roots on and outside |2| = 1 are 
denoted by sy, 51, 5, ... then the solution of (6.26) is 


Y= С, rð +2 Di s, j> 0, 


where Co, D; are constants. 
But since A v;=1, Рр=0 for all i. Thus, when p < 1, we get, 


уу = Сог, j>0 and from i yj—], we get Co 1 — го, 
so that 52 
у= (1 — ғ) n. 1270. (6.27) 
The steady-state arrival point system size has a geometric distribution 
ier mean r,/(1—r,), ro being the unique root of r(z)=0 lying inside | 
Zi [ex 1, 


Waiting time distribution: As the distribution of {vn} is geometric and 
the service time is exponential, the waiting time in the queue Wo, being 
random geometric sum of exponential variables, is modified exponential. 
We have, 

Pr (Wg = 0) = 1 — ғо 
апі Pr {Wo < t) = 1 — ro e-*Q-r9t, t 0. 


The conditional distribution of the waiting time in the queue Ио of those 
units that have to wait, is exponential with mean 1/p (1 — ro). 

Note: (1) The unique root ro (0 < ro < 1) of r(z) =z — G(z) = 0 can 
be obtained numerically by employing Newton-Raphson or other standard 
"method. : 

(2) Kingman (1963) gives an explanation for the occurrence of 
geometric distribution as steady state solution of the G//M/1 queue and 
discusses about the close formal connection between the theory of this 


queueing model and that of branching processes. 


(6.28) 


Example 6(c). The model EM]. Here the distribution of the inter- 


k 
arrival interval is Erlang-k with meam 1/à and L.T. A*(s) — ( zu) 


Thus, the characteristic equation becomes " 
Á àk L 
гада ачин) (se) 7? 


G2) lying inside | z |= 1 is denoted by го, then 


cess in Ex/M]1 
d the solution 


ч the unique zero of z — 
п (I— ry ry, n 20. | 
Моге; ie in He case of. M/Ex/ 1, the queueing pro 
Could also be viewed as non-birth-death Markovian an 
Obtained from the corresponding C.K. equations. 
Particular cases; (1) MIMI: When k = 1; 
z— K(z) == 0. gives (2 — (с — e =0. 
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Thus when e < 1, the required root r, is р and we get 
Уп = (1— р) е", nO. 
In this case v, = p; for all n (v, = p, only when Ex = М, і.е. k = 1). 

(2 D/M/l1: When k> œ, A*(s) > es^ 
and z—G(z)=0 implies ze?-2/e = 1. 

If the unique root inside |2 | = 1 is denoted by č, then 
Ип = (1— Ё) č", n0. 

Remarks: We take as regeneration points the arrival epochs in case 
of GJ/M/1 model and the departure epochs in case of M/G/1 model. The 
queueing processes occurring in the two models M/G/1 and GI/M/1 are 
non-Markovian; however by extracting processes at these regeneration 
points, it has been possible to obtain (embedded) Markov chains from 
those processes. The embedded Markov chains indicate system state at 
regeneration points. What one needs also is the distribution of general 
(or random) time system state. 

Let us now consider the queueing process of GI|M]l, having for its 
embedded Markov chain (Y;), where Y, is the System size immediately 
prior to nth arrival. We have obtained (vj), the limit distribution of {Yn} 
Define Z(t) = Yn, 1, St < ta. Then Z(t), which gives the system size 
at the most recent arrival, is a semi-Markov process having {Y,} for its 
embedded Markov chain. (Y, t„} is an irreducible Markov renewal 
process. . 

Let N(t) denote the system size at an arbitrary (general) time т. Denote 

fit) = Pr (Z0) =j | Z0) i 
Pult) = Pr {N(t) = j| NO) = i; 
then f; = Jim Sij(t) gives the limiting probability that the system size of 


the 5—М.Р. Z(t) is j, whereas p; = lim Pij(t) as t — со, gives the limiting 
' probability that the system size of the Beneral time process N(t) is /. 
We have to look for relationship, if any, existing between the three 
limit distributions (vj), {7} and (pj). 
From Theorem 7.1, we find that у, and fj are related as follows: 
TRUE... 
fi ERU a (6.29) 


where M; is the the expected time Spent in the state ; during each visit. 

The transitions, in case of this model, occur at the arrival points which 
are its regeneration points. Thus Mj, the expected time spent in a state 
i during each visit is the expected interarrival time. In other words, 
M; = 1/) for alli. Putting the value of M; in (6.29), we get 


УА yj А 
f$ REST UU for all j, 
i i 


—(-—ryri (from (6.27)). 
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In order to obtain ру in terms of f; (and in terms of vj) we have to make 
use of the relationship (obtained by Fabens, 1961) between f; and ру. 
It is shown that for this particular model, 
ру = (Ми) filro 
=(Ny)(l-r) 4, jal 
and po 1— М 


C. The Model M/G^^/1 
Model with Poisson input, general service time and under general 


bulk service rule. -As in Sec. 10.6 A, we easily see that the departure 


point process (X, = N(ta + 0)} is a Markov chain. 
Denote 4, = Pr (Number of arrivals during a service time = r} 


Тем (му 
= ar dB(t). 


0 
оо 


We have Q(s) = Z qs = B* A — As) where B*(s)— | e-"dB(t). The 


transition probabilities Ру = PF {Ха =J | X» =i} of the chain 
{Xn n > 0} are given by 
ру = Y-i > Í >P b jpi—b 


=0 ,i>bj<i-b 
eg ЕРЫ 
and =a 94162-10720. 
It is clear that the рг $ are independent of a; they are also indepen- 


dent of iforo << b and thus have the same values for all i— 0, 
1,...,5. The first (b+ 1) rows of the t.p.m are identical. The t.p.m. 


P = (ри) is given by 


ie) Ы 
ü fe h x IE 1 
1 o ГИ EE дз. 
2 qo qı d: ds: - 
P q 4 ГА da: 
por | Bie geet Bg 
TER M 
ET J 


ble and aperiodic. When the traffic intensity р 
Jess than unity, the chain is ergodic. Then the 


departure point probabilities 
, (= 0,1,2,--- 


The chain is irreduci 
(defined by р = Mbu) 15 
limiting or steady-state 


у= шл Du 
n 
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exist апа V = (ry. v...) is given as the unique solution of V = VP. 
Thus, we have, for j —0,1,2,... 


Y; = (Vo Ya ss F а) dj + Vogi + Vosa j-i + Vote di-g t +--+ Уо 
b—1 "S; 
—(Zv)9qg--(Z фу, Your). . 
r=0 r=0 
The p.g.f. of {уу} is given by 


И) = b vjsi 
Ј=0 
5-1 co ME NE Р 
=( È w) Èq s+ У(Х djor SI Vopr 57). 
r=0 j=0 j-0 r-9 


5-1 
The first term on the r.h.s. is (Х Yr) Q(s); the second term is 
r=0 


Z (a; 87) (vo) -E (45-3 Y (vus) + ... + (qo) (S5 ьа) 
= Q(s) (va + swa 4...) 


= Q(s) (1/55) { V(s) — T ws. 
Thus, (1— QS) V) = (OCICS (#— sr) v) 


5—1 
У (sb— р) у, 


ог, V(s) = SECOS ‚ As) 


3 (9—8), 
= 8700): (6.30 


Putting B* (A — As) for Qis), we also get the following expression for V(s). 
b-1 
Z (s*—s)v. 


V(s) = 0, В (А — А5) 


9—85): (6:80) 


5-1 
I (s — s)v, + 
т=0 


=B (Хх =} 1: (9910) 


Particular cases: (1) M/G/1. Letb —1, then a— 1. Noting that 


= 1—p, and putting а= b = 1, we find that the result (6.7) at 
once follows from (6.30). - 
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The dis- 


Ў abj]- 2 LY Re E u 
(2) M/M**/1: We have Q(z) = B* (A — А2) jme rame, 
tribution (rg). of the system size at departure instants in steady state for 
a :=0, 1, ..., b has the p.g.f. 


V(z):s-Z шк 
k-0 


b—1 
У (22 — 28) vk 
shay i- a tegen 6.32 
z Foi -3 1 us 
Now the denominator of V(z) has (b — 1) simple zeros z = Ry, ..., Ro-1 
b-1 
inside the unit circle | :|= 1. The b factors (= — 1) П (z — Rj), common 
і=1 


to’ both the numerator and the denominator cancel out, leaving out in 
the denominator a factor (7 — Жу), Ro being the zero outside | z| = 1 of 
the function z^ {1 ; (4/u)(1 — z)) — | occurring in the denominator. 


Thus, И(2) + and. И(1)== 1 gives: С = 1:50 


С 
5 — К, 


f 1 а Аа о) 

so that (с) = "T ad Lee” 

where ry :1/Ry is the unique root inside |=| = 1 of the equation 

whose roots are the reciprocals of those of z^(1 |- (A/u\(l — 2) — 1 —0, 

i.e. ry is the unique root inside | т | = 1 of Az^? — (à +p) 2 + p =0. 
Thus, for all a — 0, 1, ..., b, we get, from V(z)=(1 — roX1,— ro 2), 


yg == (1 — ry) б^, K 0,1,2, «ose 


For example, for b - 2, a = 0,1, 2 we get ry = (2%/0)/(1 + VU + 4/9) 
and vk=(1— л) roč; for b=1,a=0,1 we get rop, so that 
ve = -—9) e. | 

Remarks: (1) From (6.30), it is clear that the p.g.f. V(s) which is а 
function of b, is indépendent of a. Thus given b, one gets the same 
steady state departure point distribution of system size under the general 
bulk service rule for all values of a(— 1,2, ..., b), and also for a — 0, 
(1.е., intermittently available server). It is clear that though for different 
values of a, the actual processes differ, all the embedded Markov chains 
considered at the regeneration points (which are departure instants) are 
identical. 

(2) To obtain the stcady-state distribution {хуу of the general time 
system size V(t), one can proceed as follows: 

First, consider the embedded Markov chain {¥,} at departure points, 
i.e. X, = M(tn +0) and obtain its steady state distribution {v;} (this has 
already been considered in the example). 

Secondly, consider the embedded Markov chain (Y;) at not only its 
instants of departure, but also at instants when N(t) reaches the level a 
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service commences. This will imply that transitions 

a xe se 2 all instants of departure as well as a// instants of 

commencement of service (instants of departure and commencement of 

service will coincide е ies i D 3 a). Deduce the steady-state 
istribution (rj) of {Yn} from that of (vj). j 

ECC ыз а semi-Markov process Z(t), having {Yn} as its 

embedded Markov chain. Obtain the steady-state distribution (fj) of 
Z(t) in terms of {rj} using Theorem 7.2. 1 ET 

Finally, obtain the steady-state distribution {ту} of N(t) by conditioning 
N(t) with Z(t), (with the help of a theorem of Fabens (1961)—based on 
an elegant probabilistic argument concerning the number of units 
arriving between the instant of the last transition before ¢ and the instant 
t itself). 

Note that the distribution (rj) and therefore (xj) will also depend on 
a, although that of {vj} does not. 

See Fabens (1961) and for more details, Neuts (1967), Teghem et al. 
(1969). Foran alternative approach using the technique of inclusion of 
supplementary.variables, see Borthakur and Medhi (1974); see also 
Chaudhry and Templeton (1981), who obtain the p.g.f.’s of the queue 
size at random epochs and at departure epochs and a relationship 
between the two p.g.f.’s (for the M/G**/1 model). 


Some General Observations 


The pace at which research in queueing theory has progressed amply 
demohstrates its fertility. It is clear that the theory is moving forward 
and that its applicability is being enhanced. The extensive literature has 
mainly been concerned with describing the stochastic processes of interest 
in various queueing models. j 

With a view to simplify the existing theory, various methods have 
been advanced. For example, random walk approach has been used to 
provide simple proof of some well-known results as well as to obtain 
new results on a variety of queueing problems. Reference may also be 
madeto Wiener-Hopf techniques in queueing theory (Smith and Spitzer). 
While examining extensions of these techniques to continuous time, 
ladder processes have been investigated (Prabhu, 1970), Prabhu 
et al.(1971, 1973), and it is shown (Prabhu, 1974) how the techni- 
ques could be employed to simplify existing theory and to generate new 
results. Mention may also be made of the general definition and pro- 
perties of the Markov renewal branching processes which have been 
shown to be useful in the study several queueing models (Neuts, 1974). 

One topic of considerable mathematical and practical interest is the 
heavy traffic research, whose principal objectives are (i) to describe 
unstable queueing systems (having traffic intensity greater than unity), 
and (ii) to approximate highly saturated stable queueing systems (having 
traffic intensity less than but closeto unity). The term heavy traffic is 
due to Kingman who initiated work in this topic (for an extensive recent 
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bibliography, see Whitt (1974). The central theme for heavy traffic is 
limit theorems. Further areas of research are concerned with the rates 
of convergence, approximations for multi-channel queues, finite queues, 
control etc. 

As for statistical analysis in queueing theory, pioneering work was 
done by Clarke, who began by obtaining the maximum likelihood esti- 
mators for the arrival and service parameters of the simple queue. 
Important contribution on estimation of parameters and confidence 
statements about them have been made by Cox (1965), Wolff (1965), 
Lilliefors (1966). Wolff extended the work of Billingsley on likelihood 
estimation of Markov chains in continuous time, including limit theorems 
and hypothesis testing. and made use of the same to obtain results for 
birth and death queueing models. 

The research in control and optima 1 
centred around optimal desiga and control of a model with a cost struc- 
ture (for a review, see Stidham and Prabhu (1974), Sobel (1974); for a 
classified bibliography with over 150 titles, see Crabill et al. (1977). 
The decision variables considered are arrival and service parameters, 
queue parameters, operating time parameters; the objective functions 
considered are concerned with average OF discounted costs. The 
theory of Markov and semi-Markov decision processes have been used 
in various studies. Martingale approach has also been made in the 
formulation and solution of control problem in queues (Martins-Neto 
and Wong (1976). The current trend is of piece meal co adhoc ee 
Cations of general models. Stidham and jme hasise ө. h e 
need for common methodology and indicate some direction in which a 


unified theory could be developed. 


1 operation of queueing systems is 


10.7 Reliability 


A. Introduction 
ili i 's, and .the growth of the 
igation i i began in the 1950’s, an 
@ озу T momentum after a decade or so (some standard 


listed in the References). -— 
woe bd fo ial concerned with the determination of the 


y ; ill 
robabili stem, consisting possibly of several components, wi 
Рону ftat dade period RB e A 

Suppose that a component i of a system is capa ен se denoted 
of performance—functioning and not pronus he of the component i 
by | and 0 respectively- The mode oe Hh X, with values 1 
may be навї by 4 ET dior) 


dnd hethe ent is functioning or not func- 
па 0 depending on W 
tioning. Suppose that 

Pi” Pr {Xi 


then p, is the reliability of the / 


Bernoulli rand 
r the compon 


= 1} = 1 — Pr - 05 


th component. 
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Suppose that a system has some finite number п, of independent 
components labelled 1, 2,..., л, and that the system is capable of just two 
modes of performances. Represent the mode of performance of the 
system by the Bernoulli г.у X. Suppose that, given the structure of a 
system, the knowledge of its performance can be determined from that 


of its components. The system structures generally considered, are 
described below. 


(i) Series system: The system functions i 
function. We have Y — min (А, 
nents is given by 


ff all the п components 
ХХ); the reliability of п compo- 


Р=Рг{Х 510) Pr iun C. rr uy es 1} 


=PriX=1,%=1,...,%=1) (71) 


(i) Parallel system: The system functions iff at least one of the n 
components functions. We have Y - max(X, ...,Y,). The system 
reliability is given by 


Р = Pr {X == 1} -= Pr (max (X, ...,X,) =} 
=U— Prin 0 a . 12.2) 


=1— I (I — pj). 
i21 


(10) K-out of n-system: The System functions iff at least k (1 < k <n) 
of the n components function. As particular cases, we get the series 
system:for А.-н and the parallel system for k=1. Suppose, for 
simplicity, that Pi++p for all i, then X is a binomial variate with 
parameters п, р. Thus 

P — Pr (X2 1) 
=Pr {X> k) = E (7) ode (7.3) 
i=k 


Lifetime distributions: Let us now consider time. Assume that a 
device functions for some time after which it fails, and then remains in 
that non-functioning state. The duration of time T, for which it functions 
is its lifetime. A number of failure distributions have been suggested as 
models to describe the lifetime of a device. . 


Definition : The reliability (or the survival function) of the device at 
lime г is defined by' 


Rü)— Pr(T 2 th=1+ FD: 20 (7.4) 
= 1 1< 0, 
where F is the distribution function of Т. 
Denote by f(t). the p.d.f. of T, when it exists. 
Exponential distribution. which has t 


he memoryless property, charac- 
terises the phenomenon of no wear. 


Future lifetime of a device remains 
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the same irrespective of its previous use, iff, its lifetime distribution is 
exponential. But the phenomenon of wearout has also to be taken into 
account in certain cases. In an attempt io find out distributions which 
describe diverse phenomena associated with a device, certain character- 
istics have been evolved. We define below two of them. 

Definition: Hazard rate (or failure rate) function,of the distribution Е 
is defined by the ratio 


= К IO —mRu) s 

HD =т= КОРТ >} AD (ТЕО) 
for values of г for which F(t) <1. Негел(ї) dt gives the probability that 
the device has a lifetime between г and t + dt, given that it has functioned 


upto time t. We have 


t 


R(t) — R(0) exp {- | r(x) dx | 
0 


F is an increasing (decreasing) hazard rate IHR (DHR) distribution if r(t) 
is an increasing (decreasing) function of t > 0. The hazard rate corres- 
ponds to age-specific death rate in demographic studies. 

Definition: Hazard rate average is defined by 


Li 
Al r(x) ах. 
0 
F is an increasing hazard rate average (IH RA) function if the hazard rate 
average increases in г > 0. (А detailed account of IHRA distributions 
is given by Birnbaum, Esary and Marshall (1966).) We list below some 
of the distributions generally considered in the literature: 


Distribution: F(t) (t > 0) r(t) 
Exponential : ae-“ (a > 0) а 
Weibull E daite uk (а > 0, a > 0) aat*t 


a(aty-! e-at 


Gamma TG 


(а > 0, а > 0) 
Modified 3 1 { (ef — 1) } е! 
s = exp 4 — pt B. 
extreme value a a a 
а> 0 


The exponential distribution is the only distribution having cons- 
tant hazard rate, as is expected. The modified extreme value distribution 
has an IHR and Weibull distribution has IHR for « > 1 and DHR for 
O<a<l. 
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B. System Reliability : 
Suppose that a system consists of п independent components 13 й 
... n, and that the lifetime of the system is T, while that of ith compo- 
nent is 7;. The system reliability (or survival function) at timet is 
R(t) = Pr (T > 2), and the reliability of ith component'at time fis 
R, (t) = Pr (T; 2 t). 
Expected lifetime is given by 


= — | tdR(t) = ] R(t) dt. 
0 с 
(i Series system: We have T = min (T,, ..., Tn). 
Thus, R(t) = Рг {T > t} = Pr (min (7, ..., Ta) > t) 


n n 
= II Pr (7; > 1) — IL AR, (t). (7.6) 
1=1 i=l 
The hazard rate function of the system is 


r(t) = —R(0. "Ag sro» Ry © 


R(t) 


П RE 

2-1 
= "MEE EE 2 
с Rif) TATO 


where ғ; (t) is the hazard rate function of the componenti. Thus, if the 
lifetime distribution of each component is an IFR, then that of the series 
system is also so. 


- 


(й) Parallel system: We have T = max viu da). 
Thus, R(t) = Pr (T > t) = Pr (max (T,, ..., Ta) > t) 


abe Å Pr р E bi (1 — A ()). (7.7) 


For parallel system, the lifetime distribution of the System is not neces- 
sarily IFR, even if that of each component is. 


(ii) Kk-outof n-system: The System lifetime, is the corresponding 
order statistics 
PM, TO, „Тө... Те, 
(formed from 7, ..., T,) fork = 1022 a 
Example 7(a). Consider a system consisting of п components such that 


the failure of the ith component occurs in accordance with a Poisson 
process of intensity а. To find the reliability of the System under 
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different system structures. Here 7; is exponential with mean 1/2, i.e. 
R(t) = е-и, 
(i) Series system 
R(t) = II R(t) = e-Gaot, 
T is exponential with mean 1/Za;. 
(1) Parallel system 


кау=1— П (1— eu) 
i=l 


= Zeni Zeuta)! |, д eTl, 


1 1 1 
d EEPE da es, 
an ЕТ) 2 as at sy 


In particular, if а; = a for all i, then 
R() 1—(1— e«y = X (= уч( mers 
rel 
also R(t) = i ( n (e?) (1 — e-aty, 
r-1 


and ET) = (—1у— ( id ) |e. 
rel x 
By integrating in (0, 1), the identity 
n=l 
E Оре — 1. 19s, 
= 
we find that 
ETA 
Tas 
For large п, Е(Т)-> 1 (log n + y), where ү = 0.577..., is Euler’s constant. 
(iii) k-out of n-system. Suppose that а; = a for alli. We have 


R(t) = Pr {T> t) 
= X ( à ) (ey (1 а е) 


rok 
and 


E(T) =f 20 dt = 3 ( : le (1 — e)n- dt 


0 


1 
я d : 
a ( : )] Qd D E (putting у = e-*) 


~ 


n T(r) Г(в—ғ+-1) 
-krin—r) — T(n-1) 


i (x) gr, п—г+1)=,$ 


гек NF 
1 
r 
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Example 7(b). Suppose that a system consists of n components 19 25s o 
i, .... n; the first r of them, 1, 2, ..., rare all of different kinds and the 
атте (n — r) are similar but are of another kind. The system 
functions iff each of the r different components functions and at least.one 
of the remaining (п — г) similar components function. Suppose that the 


failure of the ith components occur in accordance with independent 


Poisson processes, with parameter a; fori—1,2, ...,r, and with para- 
meter b for i- г | 1, ..., n. 
Then 


R(t) = e7", for i= l, ...,r 
agh for ist FR et 
so that 


no ={ tt no) i-a - RO) | 


= TI en ait (1 — (1 — е-Мут}. 


1-2 


Markovian Models in Reliability Theory 

In Example 7(a), the failure process for individual components is assu- 
med to be Markovian. The assumption that the failure process and the 
repair process, whenever repairs are considered, are Markovian is very 
often made; in such cases it is possible to write down at once the 
Chapman-Kolmogorov (C K.) equation of the Markov process of the 
resulting system. From the solutions of these equations, one can find 
the system reliability. The following example will illustrate the method. 


Example (c). To find the reliability of a system consisting of n identical 
components connected in parallel, such that the failure of a component 
occurs independently of the others and in accordance with a Poisson 
process with intensity a. (This reduces to Example 7(a) (ii) with a; = a.) 

The operating process of the system may be represented by a Markov 
process {X(t),f>0} with (n-- 1) states 0, 1, ..., 7..., n, the state i 
denoting that there are i failed components at the instant t. The state п 
is an absorbing state. The transition to state п denotes its first passage; 
and transitions from any of the other states to some other states avoid 
the state л. 

Let p(r)--Pr(X(t)— i) and the initial conditions be p,(0) = 1, 
pO) =0, i250. The reliability R(t) of the system at time ¢ is given by 


R(t) = pe(t) + p(t) +... +Pn—(t) = 1 — ХОР 


It is clear that the process {X(t}, > 0) isa birth process with state 
dependent rates 4; given by X; =(n—i)a,i= Ty uci m being an 
absorbing state. 

The C.K. equations of the system are thus given by * 
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Po (t) = — na p(t) 

pi (t)= —(n— i) a p(t) + (n-- i+ Da pit), i—-1,2,...,n—1 

p» (t) = aPns(t). 
The first equation can be solved at once and the other equations (which 
are linear) can then be solved recursively, Alternatively, the equations 
can be solved by considering their Laplace transforms. Denote the L.T. 
of p,(t) by P;(s). By taking the L.T. of the above equations and making 
use of the initial conditions, one gets 

sP,(s) = 1 — na P,(s) 

sP(s) = — (п — i)a P(s) + (п — i+ Da Рі (5), i=1,2,...,2-1 

5P,(s) = a P, 4(s). 
For the particular case, л == 2, we have 


1 


A =ош 
апа | 
2а Е 3 ) 
һ®= S Ti) EX. -—( Far 
Taking the ЗС L.T., we get 
Po(t) = ет“ 
and Pit) = erat — gat); 
Hence R(t) = pilt) рЫ) = 279 — et. 
In the general case, we get Se 
1 к. 
Руз) = з Епа 7 
—i+ la T a 
Ps) = ace Wa Pid, t= узек 
Ps) = Z Р,_1(5). 
Thus, Р) =f s (8) = — d. Profs) 


sta 


Ip 2 
55 +а).. (= па) 


Resolving inte partial fractions, we get 


a(2à)...(d) А, 4 А _ An 
s(s- a)... .(s nay _ S aha пани E s+ nd 


22(45-95/1982) 


55 
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where 4o ... are constants, which can be determined as follows. Multi- 
plying both sides by s and letting s— 0, we get Ag B 
Multiplying both sides by (s + іа) and letting s — — ia, we get 
p a-(2a) ... (na) 
i —(—ia) С G— Na). ..C a) (а)(2а)...(п — )а 


E per DG REED (ay (7). 
n n n 
нь Сезш (= d. f e 


Inverting the L.T., we get 


pat) «i-(1)e7 СЕЕ shp) 


== (1 ery. 
Thus, R(t) = 1— рй) = 1 — (1 — e7“, 


Note : This result was obtained in Example 7(a) by the straightfor- 

ward method which was much simpler. This alternative method is given 
only to illustrate the application of the theory of Markov process. 
We now make the assumption that a failed component can be repaired 
and renewed. The system can then be described by a queueing model, 
and the methods cf queueing theory can be applied in its study. То 
find the reliability R(t) at time t, one needs the transient state probabilities. 
We consider an example below. For more general discussion refer to 
the standard texts on Reliability theory, such as Gnedenko et al. 


Example 7(d). Parallel system with maintenance. Consider a two- 
component parallel system, with a single repairman available for main- 
tenance. Suppose that the failure and repair processes are both Mark- 
ovian with rates a and b respectively (see also Exercise 4.11). 

Here the process (X(t), t > 0), has 3 states i = 0, 1, 2, the state i denot- 
ing the number of failed units at the instant £. "The state 2 is an absrob- 
ing state. This may beconsidered as a machine interference problem 
(see Sec 10.4); the process may be represented by a.birth and déath 
process having 3 statesO, 1, 2, the state 2 being an absorbing state. Let 
pi — Pr (X(t) — i); the C.K. equations are given by 

po(t) = — 2a po(t) + b рї) 
P(t) = — (a + D) p,(t) + 2a p«(t) 
PK) = a p(t). 
Let the initial conditions be p,(0) = 1, p;(0) =0, i = 1, 2. 
The above-equations can be easily solved by taking their Laplace trans- 


forms. Denote the L.T. of p(t) by P(s). Taking L. T. and using initial 
conditions, the first two equations can be written as 
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(s+ 2a) P,(s) — b P,(s) = 1 


— 2a P,(s) + (a +b + s) Р, (5) = 0. 
Solving, we get 


P 2a = a+b+s 
PO- эк Gat sta’ 90) =a pap ` 
P, (s) can be written as 
2a . 
АӨ = лушту ыу 
vilis $  — Ga+b) + vie +b) + 4ab} 
S2 


Thus, 


dpn SL cL 


$,— Sg\S—S, S— Sp 
and its inverse is 


pitt) = 22 (eta eno 
Again, PAs) "Ves 


2 1 (taa tics 
(S—s)\ s—35 S— 5; 

and its inverse is 

pot) = CEED ev —G@t+h+ Hem 

$3 — Sq 
The system reliability is given by 
R(t) = p«(t) + p(t) 
Ga +b + s) en! — (3a + b + sa) e 
(5, — 5) 

Using the relation s? + (За + b) s + 222— (s —s,)(s—s,), R(t) can also 
be expressed as 


t conf 
RQ) =% en Sge 
© (51 — S2) 


C. Sheck Models and Wear Processes 

We consider here a model described by Cox, Esary, Epstein, Gaver and 
later exbaustively examined by Esary, Marshail and Proschan (1973) for 
the lifetime of a device. Herewe give some of the interesting results 
obtained by Esary etal, to which references may be made for 
proof of the theorems stated. Some of the results of Esary ef al. have 
been generalised to cover the multivariate case in a recent paper by 
Marshall and Shaked (1979). 

The device is subjected to shocks which occur in accordance with a 
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Poisson process with parametera. Denote the probability that the 
device survives the first k shocks, (k = 0, 1, 2,...) by qe. Clearly, the 
reliability of the device at time f is 


R (t) — Pr (T t} = 24 ета (at elk, 15 0 (7.8) 
= 1 PE, 
Now, ак being the probability of surviving k shocks, we have 
124 24 24... (7.9) 
and рк, the probability of failure on the kth shock, is given by 
Pe=%1—% »k=1,2,... (7.10) 
and n=l- = Pr=1— qs. A 


The p.d.f. f(t) of thc lifetime T is given by 


AD = – R= a E реа) — 1)! £0 (7.11) 
k=1 
and Pr {T= 0} = 1 — lin | fe) dt = po, t=0. 
£20 4 


Its hazard rate function r(t) is given by 


cof 


=a Е = { 5 ЖОШ 3 Gk оча}, ї>0 
к=0 k=0 
so that r(t) « a. 
The equality holds if д, = 
has exponential distribution. 


One can express gy and ру in terms of derivativ 
Denoting by R (1), the kth derivative of RO), 


D 5 RO J 
a= 3 (^) n уш, 


0 forall k > 0, ie. if q= 1 гамі then T 


es of R(t) as follows: 
we have 


к 


k—1 
Рене » j 
рь = Pd D + RO O)/ai, 
The jth moment yj, j= 1, 2, ... of T is given by 


е Ikpi—] 
T "ui ) ae 


Provided the series on the r.h.s. converges. Some of the properties of 


Фо of interest in reliability, carry over to R (rf). One such property is as 
given below. 
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Theorem 10.1. R(t) is IHR if q;/qy-, is decreasing ink = 1,2,.... 


Models for дь : Consider that damage is caused to the device by a 
shock. Suppose that the ith shock causes a random damage X; and 
that damages are additive, so that the damage caused by the first k 
shocks is X, + X,+... + Xy. Suppose that there is a capacity or 
threshold which the device can endure. The threshold may be consi- 
dered to be a constant or to be a random variable Ё. We assume here 
that the capacity or threshold has a fixed value x. The device survives 
the Ath shock if X, + ... + X, does not exceed the threshold value x, 
so that 

Qe = Pr(Xi +... +X, < x}. 


Now the nonnegative random variables X, may be (i) i.i.d., (ii) indepen- 
dent but not identical (as successive shocks could cause increasing 
damage) and (iii) neither independent nor identical (as accumulation of 
damage may result in loss of resistance to further damage). 

If X, i = 1,2, ... are independent but not identical (and have F; for 
their d.f.), then д; is given by their convolution 


gy =R f osx „Р 


If Y, are iid with d.f. F, then the d.f. of X,+ ... + Xk is the kth 
convolution F** of F, k = 1,2, .... Then 


qe = F** (x), (x fixed), k —20,1,2, ... 
0$ au (at) 
and R(t) = e-at SED F** (x). 


The process represented by R(t) = R.(t), which has x as a parameter, 
is a compound Poisson process. 

The stochastic process {Z(t), t 2 0) representing the accumulated 
damage or wear in (0, t) is called the wear process. For each t, 


Pr (Z(t) < x) = Rx(t) = R(t). 
We state below an important theorem concerning Z(t). 


Theorem 10.2. Assume that Z(t) satisfies the following conditions: 

(i) {Z(t), t > 0} is a Markov process, 

(i) Z(0) =0, Z(t + 5) — Z(t) > 0 for all t, h > 0, with probability one, 
and (iii) Pr (Z(t - ^) — 20) <и | Z(t) = u} is decreasing both in Z and 
t for t > 0, h > 0, u > 0. 

Then the first passage time Т, = inf (t: Z(t) > x} of the Markov process 
Z(t), is IHRA. 

Note: (1) The condition (ii) means that the device enters service at 
time 0 with no accumulated wear, and that wear is non-negative and the 
condition (iii) implies that, for fixed г, an accumulation to wear can 
only weaken a device and make it more prone to further wear and that 
given equal amounts of accumulated wear,a device older in age is more 
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prone to additional wear than one younger in age. The three conditions 
seem to be fairly realistic. 

(2) Compound Poisson processes are included amongst the Markov 
processes satisfying the conditions (ii) and (iii). 

(3) When the threshold has a fixed value x, the first passage time Tx 
of the Markov proeess Z(t) is equivalent to the lifetime T'of the device. 


Example 7(e). If 
к= 1, К< п 


=0 k >m 

then from (7.10) we get 
рк=1,К=п-+ 1 

== 0, otherwise; 

from (7.11) we get 
y~at л 

д) = 5080, 
so that T has a gamma distribution; in particular, if n = 1, i.e. @= 1, 
qx =0, k = 1, 2, ..., then T has an exponential distribution. 

Hammed and Proschan (1973, 75) have extended these shock models 
and wear processes to cover the cases in which shocks occur in accor- 
dance to non-homogeneous Poisson process and more general birth 
processes. 

The discussions in this section would give an indication of the role 
of stochastic processes in reliability. Tkere are various other models in 
reliability where interesting stochastic processes arise. Cinlar (1975) 
shows how the theory of Markov renewal processes could be applied in 


reliability. 


Current Trends: Some surveys 
.The qualitative concepts of. reliability are somewhat old but its quanti- 


tative aspects have been developed over the past two decades or so. 
The developments are quite varied; here we mention only a few recent 
survey papers of interest. All of them contain exhaustive and useful 


lists of uptodate references. 
Barlow and Proschan (1976) ind 


lists academic research in reliability. 
Reliability, which has long been considered as a measured of system 


effectiveness, is now considered as an incomplete measure. Availability, 
which is a combined measure of reliability and maintainability is now 
receiving wider attention and usage as a measure of maintained system 
of efiectiveness. An elaborate survey of the literature with a classification 
of the problems is given by Lie et al. (1977). e. E 
Considerable attention has been paid to reliability optimisation. A 
special issue of IEEE Transactions, Vol. R-26 (1977) is devoted to this 
topic. Tillman еѓ al. (1977) present a review of the theoretical optimisa- 
tion techniques used in reliability, and provide an extensive bibliography. 


icate some directions of current 
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10.1 


10.2 


10.4 


10.6 


EXERCISES 


Consider the simple queue and let N (t) denote the system size 
at time г. Find the transition probabilities of the Markov process 
{N(t), t > 0}: 

pu (At) = Pr {N(t+ At) = j| NO = 0. 
Write down the rate matrix aud hence derive the. Kolmogorov 
equations. 
Find the distribution of the number (Q) in the queue in steady 
state in a M/M/1 queue with rates A, p. Find E(Q) and 
var (Q). Verify that E(Q) = ^E (Wo). Find also var (Wo), 
and verify that var (Wg) + var (v) = var (Ws), where v isthe 
service time. 
Obtain the distribution ofthe number in the system in steady 
state for M/M/s model by considering it as a birth and death 
process. 
Using Little's formula obtain the average waiting time in the 
system for M/M/1/R model. 
Centralisation of counters 
Consider that two identical M/M/1 queueing systems with the 
same rates à, p. (intensity р = Х/ы) are in operation. Show that 
the steady state distribution of the number N in the two systems 
combined is given by 


Pr {N = n} = (n + 1) 0 — 9e п > 0. 


Now suppose that it is decided to form a single queue instead of 
two queues but to retain the two servers. Show that the steady 
state distribution of the number M in the system is given by 


Ро) ee 1 


SS ,n-Q. 


+ 
о 


Compare some measures of effectiveness and performance of the 
two models. 

Output process of a Markovian queueing system й 
Show that for а ММ] queueing system in steady state, the 
intervals between successive departures are i.i.d. random variables 
having the same distribution as the interarrival interval. 

Show that the output distribution is identical to the input distri- 
bution also for the M/M/s system in steady state. (In other 
words, the output process is identical to the input Poisson process 
for a M/M/s (s> 1) queueing system in steady state) (Burke, 


1968). 
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10.7 


10.8 
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Let ' i 
eU 
р(К, ғ mma 


E (s, r) => р (К, r). 


(Tables of these functions аге availables.) 
Show that Е (Wo) for an M/M/s queue in steady state can be 


expressed as 
a _ „2060 17 
Ewa = |е 010009} | 
р = fps = rfs. 


Shonick (1970) refers to some studies on hospital administration 
and reports the following conclusion generally arrived at in 
regard to occupancy of hospital beds: 

When the number of beds is such that all the beds are rarely 
filled up (such a situation arises in emergency wards for non- 
scheduled and emergency admissions), then one may expect the 
distribution of the number of occupied beds to be Poisson ; on 
the other hand, when the beds are often all filled, Poisson does 
not give a good fit. 

Assuming that demand for beds occurs in accordance with a 
Poisson process and that occupancy times are exponentially distri- 
buted, give a plausible explanation for the occurrence of the 
observed phenomena. 

Combination of a loss anda delay (waiting) system (Shonick, 
1970). 

Consider that two types of patients arrive at a hospital having a 
fixed number of beds, say ғ. These are (i) elective customers (or 
physician generated) and (ii) emergency customers (nature gener- 
ated), their arrivals being in accordance with Poisson processes 
with rates a, and a, per day respectively. Assume that the 
duration of hospitalisation of both types of patients has exponential 
distribution with mean 1/b day. Elective customers who donot 
find a bed vacant wait and form a queue, while emergency custo- 
mers who donot find a vacant bed leave and are lost. 

Show that, in steady state, (when a, < bs), the distribution of the 
number in the system N is given by 


by 
E 2 nzly2,...8 


Рг (№ = п) = р, = 


з byi-s 
tae n—541,542,... 


where 


3 [z (a]bY +0 ( = JE a — a, + as. 


r! 


r=0 
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10.10 


10.11 


10.12 


Find the expected number of patients (i) hospitalised and (ii) in the 
queue. x 
Obtain the corresponding results for M/M/s/oo and M/M]s/s 
modeis as particular cases. 
Consider a M/G/s/s-loss system in equilibrium. If the random 
variable N denotes the number of busy channels and 
B(s) = B (s,Muy = Pr {N = 5), then show that 
E(N) = (A/p) (1 — B(5). 

Suppose that x; is the indicator variable of the ith randomly 
chosen channel; x; = 1 or 0 according as the ith randomly 
chosen channel is busy or idle; and that p,(4) denotes the probability 
of the-event A (with reference to a M/G/s/s-loss system in equi- 
librium). Then show that 

Ps Qa = 1) = (Мы) (1 — B(3)/s 

0 1, ras xem 1} 

= B())Bs-EF, 1<k<s 
Ps {ху = 1| X= 1, X4 1, lusu Xe = 1} 
(/p) [1 — B(s — Ю)]/(з — К) 
= p. = 1) 

(Kaufman, 1979). 
Machine interference with m machines ands repairman. (a) Show 
that the mean arrival rateA' is given by 


х = У (т — n) 3p, = ^ (m — L) 


n 
T MET 
X Am) 
(b) Denote by X, the expected number of machines in working 
order. Show that, when s is close to m, then 


and that Ww 


m 
Хе —— 


1+A/p 
and when s is small, then 


5 
Хе 
Oe) 
(c) Special case: s = 1. Show that the probability p, that the 
number of machines in working order is r is given by 


oc rupem a 

Dur coa КҮ 
A mechanic looks after 8 automatic machines, A machine breaks 
in accordance with a Poisson 


down, independently of others, i r | 
process, the average length of time for which a machine remains 
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10.13 


10.14 - 
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in working order being 12 hours. The duration of time required 

for repair of a machine has an exponential distribution with mean 

lhour. Find: 

(a) the probability that 3 or more machines will remain out of 
order at the same time; 

(b) the average number of machines in working order; 

(c) for what fraction of time, on the average, the mechanic will 
be idle; and 

(d) the average düration of time for which a machine is not in 
working order. 


Multistage Poisson queueing system with infinite numbér of servers: 
Jackson et al. (1980) have considered a finite number k of M/M/co 
queues in tandem as a mathematical model for patient progress in 
а certain disease. Let the arrival to the first stage be Poisson 
with mean rate 2, the service time at ith stage be exponential with 
mean 1/u;; let the probability that at time t there are 7; customers 
in stage i (i = 1, 2, ..., k) be P(t, ns, ..., пк, t). Show that the 
differential equation of the system state is given by 


OP (n, п,, ..., ng, É) 
et 


k 
= -( + 2m и) P(n, ny, s is P) 
+ 8(п,) ХР(п,— 1, ns, ..., ng, t) 
ka 
+ i 8(щ+,)(т + 1) ш Р(т,, ..., 0; 43-1, п — Lu... Ne Т) 


T (nk + 1) Ek P(n, Dg, ..., M+ 1, t), 
where $( —1, n20 
=0, n=0. 
In particular, when k = 2, show that the equation admits of the 


solution 
f єч ea а" e? аз" 
P(n, ns, t) { mi H nj } 
where а, = (Ми) (1 — ек) 
а, = (Mu) (1 — eH — ut eH). 
Deduce the steady state results. 
M/M**/1: Busy period distribution. 
A busy period starts at the instant the idle server takes a batch for 
service leaving an empty queue and ends at the instant the server 
completes service of a batch leaving, for the first time, r units 
(0 < г < a — 1) in the queue. 
Denote by f(t), the p.d.f. of the busy period ending with r units 
in the queue, r = 0, 1,2, ...,a— 1. 
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10.15 


10.16 


10.17 


Show that 
1 
Г) = i [wo | exp ( — (ut + Au)) E = D 
k=0 ! 
0 
ka+r ka+r--1,6 
E HE +r (2%ри) = pear kart] (20 | аи 
where ph = uh, 
and n (х) = i (х]2у*+т\т+з) [1/n! (mn + 8)], 
n=0 


s being a positive integer. 

Obtain the busy period distribution of the simple queue M/M/1 
as a particular case (Neuts, 1967; Borthakur, 1975). 

The M/M%/s queue (s-channel Markovian queue under the 
general bulk service rule). 

Let P,,, denote the steady state probability that r channels are 
busy and n units are waiting in the queue. Find the difference- 
equations of the system in steady state. Show that 


Рп = Aro n20, 

where А is a constant and ғо is the unique root in (0,1) of the 
equation 

A= (A+ sp) x +s uxt = 0. 
The system M/G%2/1 (Single server system with Poisson input 
with service in batches of fixed size а). Let Xn Yn, Z(t) denote the 
same processes as indicated in Sec. 10.6(c). 
The Markov chain {X,} has the steady state distribution {ук} having 
p.g.f. as given by (6.31) with b =a. 
Show that the Markov chain {Y,} has steady state distribution {re}, 


where гк = Сур, ica 
, =c(v +v), k=a 
= сук, k>a 
a-l 
and v= Po Vj с= 101+ v). 


The semi-Markov process {Z(t)} has steady state distribution 
fy = lim Pr (20) = j} = уйш 
t>o 


шу being the mean waiting time in the state j (Fabens, 1961). 


M)/G2:4/1 model: Suppose that customers arrive in batches of 
size г > 1, the probability that a batch of r arrives in an infinite- 
simal interval of length dt being Acrdt (Z с, = 1). Suppose that 


r 
service is in groups of size k(a <k <b), and that n(x) dx is the 
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conditional probability that the service will be completed in 

(x, x + dx), given that it has not been completed till time x. 

Let 

V,(t) = Probability that at the instant ‘£’, the queue length 
is equal to r and the service channel is idle (it is 
non-zero only when 0 r a— 1); 

U,(t) = Probability that at the instant ‘7’, the queue length 
is equal to п and the service channel is busy 

(п 2 0); 

U,(x, t) dx = Probability that at the instant ‘£’, the queue length 
is equal to © and a batch is being serviced with 
elapsed service time lying between x and 
x-+dx (n>0), so that 


U,(t) = | Ох, t) dx. 


Obtain the differential-difference equations governing the system 
in terms of the above. Deduce the steady state relations. 
10.18 Average cost in a batch service queue: 

Suppose that units arrive at a service facility according to a 
Poisson process and that the single server isan infinite capacity 
batch server. When the serveris free to serve, and there are 
fewer than g (> 1) units waiting, he does not start service; but 
if there are q or more waiting, the service of all those waiting is 
initiated in a batch. 


Let à = rate of arrival, 

— service time of a batch, 

— fixed cost of each service, 

— waiting cost of a unit per unit time, 


— length of time between the start of nth and (n 4- 1)st 
batch services, and 


> 
> точ 


Y, = cost incurred during nth service. 


Show that {X,, Yn} is a renewal reward process and that the expect- 
ed long run expected cost C(q) per unit time is given by 


EQ E AME) 02) S p, (i — дт +n) 
40) =Е(Х)= 


9—1 
Е) + 2 р, (4 — пух 


where p, is the probability that exactly n units arrive during a 
service period. . 
Deduce the expected waiting time per customer (Weiss, 1979). 
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10.19 


10.20 


10,21 


Consider a k out of n-system, each of whose components has an 
exponential life time distribution with mean (1/а). Show that 
the variance of the life time of the system is 


(Ма?) X. (үг). 


The lifetime of the ith component of an n-component system has 
an uniform distribution in (0, г). Find the reliability of the 
system when the components are connected (i) in series, and (ii) 
in parallel. 

Consider an equipment consisting of two components connected 
in parallel. Each component has a failure rate a when both the 
components are operative and has a higher failure rate ca (c > 1) 
when either of the components has failed. Find the reliability of 
the system, and also its mean lifetime. 


10.22 Two-state process (see also Example 5(b), Ch. 4). Consider a 


10.23 


system consisting of a single component that can be in two 
states, functioning and non functioning. Suppose that the failure 
and the repair processes are both Poissonian with rates a and b 
respectively. Find the probability that the system is in an accep- 
table state at the instant f given that the system was fully opera- 
tive at t= 0. 

This probability indicates the availability of the system, and is 
denoted by A(t). 

System reliability in a random environment 

Consider a parallel system of n components, situated in a random 
environment such that 

(i) shocks occur according to a Poisson process with parameter A, 
(ii) when a shock occurs, each component faiis, independently of 
the others, with probability p(— 1 — q). 

If the interval Z upto the first failure of the entire system has 
d.f. H (t) = Pr {Z < t), then show that 


H ()=1—H@ = X c» (7) exp -A0 -9- 


Suppose that the condition (i) is replaced by (ia) : 
(ia) shocks occur according to a renewal process having interarri- 
val distribution with d.f. F(x), and with mean 1/A. 


If H* e| e (1 — H(t))dt and f*(s) -| е" dF(t), then 
0 e 
show that 


НЧ) = Ae уз (>) БӘ 


Show that the mean time pn until failure of the system is 
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„=й 3 c y (2)Ja — o». 
(Rade, 1976). 

10.24 Consider the same situation as described in Exercise 23 but subject 
to the following replacement rule: The system is exchanged before 
failure (as a preventive mcasure before failure) if the total number 
of failed components exceeds k and is replaced upon failure iff all 
the components fail and otherwise it is left alone. A cost c, is 
incurred when the failed system is replaced and a cost с,(< су) is 
incurred when the system is exchanged before failure. 

Show that the meantime that the number of failed components 
exceeds k for the first time is 


Eo ENS nens] 


Verify that this is equal to the meantime to failure of a (k + 1)- 
out-of-n system. 

Show that the expected cost per unit of time for an infinite time 
span in the steady-state is 


c(i) = +692 cofe [i)e (1) a-et] 


е e e 
ет та sey nl. 


(Nakagawa, 1979a). 

10.25 Consider a parallel system of n(>2) components in a random 
environment which generates shocks at a mean interval of (1/A). 
Suppose that the damage caused to a component is cumulative 
and the probability that a component falls by ith shock is 


p()im12,.... It PG) 3p) and 7(j) -1— P(j), then 


show that the mean time until failure of the system is 


m= (5) £c o (T) £won. 


Obtain pn (of Exercise 23) as a particular case from the above. 
[Hint: put р (i) = 41р, i= 1, 2,...] (Nakagawa, 1979b). 
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(1 — e~)?/s?. 
@ GtG () G+G(— DS 
(iii) Cy + Can + C42". 
(alp* — ai 

(py —1 5 "16 
Three eigenvalues are 1, (— 1 + 4/5)/4; the corresponding left 
eigenvectors are (1,1,1); ((—1— 5)4, (— 1 + 4/54, 1); 
((— 1+ №5)/4, (— 1 — V5)/4, 1). 
(i) Imprimitive, (ii) Imprimitive, (iii) Primitive, (iv) Imprimitive, 
(v) Imprimitive. 


Chapter 3 
Poo = Pur = Р, Por = Pio = 9; 
Poo =p =p + 4% pR = Pie) = 2р4. 
p> = pt? = p(p? + 349; p? =p = q(3p* + 43). 
Pr {X,=0| = 1} = n, Pr {X% = 1 | X, =0} =p, 


ру = 0, py =4, KAJ 


Pr {X,=1| X% = 1} = р = 4, Pr(X,22] X,=3}= 
Pr(X,—3|23,—2)—34 


When the number of children is m(> 3), P will be an m x m 
matrix such that pj; = 0, pj. = 1/(m— 1), jÆ К. 
р q 0 0 


0 v0-—p ү 
P-ip aq 0 opP'-—("pa, pa, 9%), m= 2,3,.... 


0 70 B а 
P is a (6 x 6) triangular matrix (ру) such that 
Pii —jl6, Pik = 0, k «J Dip = 1/6, k>j. 


) 
For P*, руу = (j/6) pt? = Iippa—0, К<] 


р = = Хр рк = (2k — 1)/36 
Pr(X,— ^ — 91/216. 
= (рук) is a (3 х 3) matrix such that Pij =з, 
Phin = (3 — /)/3, Prk = 0, KAI j +1. 
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The eigenvalues of Р аге t; = Ј/3, j = 1, 2, 3. Left eigenvectors 
corresponding to these eigenvalues are (1, 0,0), (1, 3, 0) and 
(1, 1, 1) respectively; the right eigenvectors are (1, — 2, Hy 
(0, 1, — 1)' and (0, 0, 1) respectively, so that c, = 1, c = 2, сз = 1. 
Thus 


= i сіу Bj, Р" = Ў спу Bj, 

=1 ~ Ј=1 
1 .(0 2 -2 
о) B,—|0 1 =l} 
0 0 0 0 
1 
1) 
1 


lim Р" (0, 0, 1), as п-> oo. 


j= 
where B, = (i 


w 

5 

ll 
араада 

os © 
ooo OON 


3.6 Forr cells, Р = (p;x) is a (r >: ғ) matrix such that 
Piy = jlr, pia = (т— lr рук 0, KAJ, ј 1. 


ilements of P? are (212), 
where p? = (lr, PPa — (г —/) Qj DIP, 
р (J) =Т= Um, 


po = 0, ks jj 1 +2 

The eigenvalues of P are t; Ser Je 1,3, tay fe 

The right eigenvectors x; of P are solutions of the equations 
(P—t) 1) Xi 20, Xi = Qi «Xi i г) 


The solution of the homogeneous equations (which сап be obtained 
by the method outlined їл Jordan,{ section 173) is given by 


(СЛ tm 


»=( 1) 


к= бе 


[rs ; 
Similarly y= ( — 1 * C = :) s<i 


whence 1/с; = 1 | ( à ) Thus P and P" can be written down. 


It follows that 
lim P^— (0,0, ..., 1) as п %. 


The interpretation of the limit is intuitively clear. 


TCalcuius of Finite Differences, Chelsea, New York (1950). 
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3.7 
3.8 


3.9 


3.10 


4.3 
4.4 
4Л 


4.9 


4.10 
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(i)-(iv) Irreducible; (v) Reducible, has two pairs of closed chains. 
2n 
(n) _. n 
i =o )r a 
Using Stirling's approximation for n! (п large), we get 


Ph? == (Apgy'| vV (пт); 


when р =}, 2pG =z EC diverges but р" —> 0. 


Hence the state 0 is persistent null if p = 3. 


When p +}, pg c а", a=4pq<1 


and so Zp? converges; thus the state 0 is transient. Similarly 


for the state 1. 


Let ff? be the probability that starting from state j, the state k is 
reached for the first time at mth step. We have 
Л =1—а„ п=0 
1010,51 E —G,),. B= 1, 2,... 
Foy = Bf = (1 — a.) + Z a...an- (1 ~ an) 
= 1 — lim (a,...a,). 
Thus the state 0 is recurrent iff lim (a... .a„) = 0. 
We have 
Hoo Z( ff =1+ È а... 
п=1 
Hence the state 0 is positive iff 2 a,...a, is finite. 
n=1 
The other states 1, 2,... will be recurrent (or persistent) and non- 
null (or null) according as the state 0 is. 
To proceed as in 3.9. 
Chapter 4 
exp Qu(s — D/(1 — qs)}; Mlp, Xt (1 + 9)/р?. 
28; 42 
р, (t) к= пир„(ї) + (a T Пыр, (t) 
E (N(t)) =i exp (— №) 
var (N(t)) = i exp (— At) (1 — exp (720). 
i 
1, when u >A and (8). when л> y. 


p.) – (QE n8 pÁO--Q0(0—1) E a)p, (1) ri 1)epasalt), 
n2 


B) = — apt) + рї) 
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As t tends to оо, M(t) tends to a/(u. — A) for p > A. 
TY nm—iXi1 1*'1 
41 =(< — pena m e 
2 mm-()i cv"(Ci)ieiE: 


443 E(Z)- (п + 1). 


- 414 (i) 0.458, (ii 0.956. 


со 


ш. ii d f eM — eN} pee ut = E 16 Tix ) 


(Note that JV does not include the parent.) 
418 Z is négative exponential with mean 1/2 ;; for i = 1,2, ... n. 
449 When Хоу œ, (vj) defines a probability distribution. 
When Х a; = œ, v; = 0 for ali /. 


4.22 — 2а а а 0 1 
А = b, — (a + b) 0 a 
b, 0 — (a+b) a 
0 b, b,  — (b, + ba) 
V = (kbjb,, kab;,, kab, Ка?), 
where -1 — (а + b,)(a + bo). 


423 v =k, v, = (ajb) k, vs = (ajb)? k, where 
k = [1 -+ (a/b) + (a/b) T; reqd. prob. = vo + Yr 


Chapter 6 
64  L*(s)— {1/5 + BY) + {ab + AQA +B)/s 
+ (ab(4A + B)}/s* + 2(ab)?/s°], 
A=pa4(1—p)b, B=(1—p)a + pb. 
10)= È Кай), 
=1 
where 
K, =A, K, = ab + AQA + B), K, = ab (44 + B), К, = 2(ab* 
a, = te, a, = (1 — e~8)/B? — te-?'[B, 
ГА = е! + g-B)|B* = 2(1 =% е—Вг\/ BS, 
а, = (3/28) — (2 + e IB + 301 — eB) BS. 
613 M(t) = abt{(a-+ D) + {ella + by) (1 — exp (— (a + b) 0) 


муд = abil(a +5) + (bla + D) {ехр(— (0 +0 Hs 
AM(T) B ,A 0\1 о) yr) 
аме Ed MA 2p? } : 


d variance of the lifetime distribution), 


6.15 


(и, o? are the mean an 
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7.1 


7.2 


73 


7.4 


7.6 
7.10 
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Chapter 7 
@ fj (s) сап be found from 
F* (s) = [I — C*()]7 4* (5); 


2)? 
0=8°+7639 


NES мыш +3) 
1 p". 

3 7 4 
f= Ber 


2 
fi. HED ELD 


where 
A, = 35° + 22s? + 52s + 37. 


On factorising Ау, then splitting into partial fractions, and finally 
inverting the transforms, expressions for / (1) can be obtained. 
(Ш) fy = 28/37, f, = 9/37. 
O ga) = (2/3) e + е + 21e — ec 

в) = e + Ие" 

Ealt) = (3/4) е7" + (1/4) et 


3e-9 | Зе-їЗ_ 3e- 
== 4 — — —MM 
s. (72e f s t*— d } 


Using (5.6), expressions for g7 (s) can be found. 


Note that / (s) сап be found also by using (5.4) and (5.4a). 


(üi) tu = 37/36, Hoy = 37/24 
(Ш) раа 7/3, Bar = 3/4. 

Gi) ра = 7/6, Qao = 23/7 
(ш) fo = 49/187, Л, = 138/187. 


The relation can be found by simple probabilistic argument 
Taking L.T. of the relation, and multiplying both sides by 5 


(fh (8) = (веб (8) 0 (5)} 
and then taking limits, as s — 0, опе gets the relation e; —/7/ Му. 
var (Vip) = 17/9, var (V3) = 13/16. 
ды (t) = 0, i2Lhjci—l 

t 


-[ or a0» t> 1) 1—1 


0 
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£T аву), 7 >0 


дый) =[ ü — n 


711 Qy (t) [а е-ач-»›} £7 02" аву), j>0 
n 


M 
Qn) (= f e SP аво), 1<i<c,j>0 
0 
= 0, i>oj<i—e 


1 


е-%(ау)і-і+е y " 6 
Е, Geico! dB(y), i2o6j2i-c. 
0 


7.12 Equality holds in the former case and does not hold in case of 


the latter. 
Chapter 9 
9.2 (М5 — 0/2 
9.6 age АЎ, ге, 2: 
9.9 н) = LM ард s) 
9.10 be = (1 — q]pY(alp-, k= 1, 2, ... 
CT EN as 
— k 
ROSZ bys* = S 


n gn-lg — 
PUG D 5а 


932 Ò m (qu pei qn — p) 


= l/a (n+ 1), р = Ф n= 1,2,3, .... 
" So (1 — so) m" (m — 1) 
Gi) no euim sym) nt! 
USOk „malhami, 25 3. 


= TF cm DH +e – 2)) 
944. vara) CTH exp (00) exp WOY П 


if u’ (1) 40 
= u” (Dr ifu (1) = 0. 


ы 


“ 


Pr(A); Pr{A} 
r.v. 


w.r.t. 

].h.s. (r.h.s.) 
XY 

FySF, 

Fae 


A 


em 
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ABBREVIATIONS 


probability of the event A 

random variable 

probability density function (or density function) 
probability generating function 

probability mass function 

cumulative distribution function 

if and only if 

identically and independently distributed 
expectation of the r.v. X 
variance of the r.v. X 
Laplace transform 
Chapman-Kolmogorov 
transition probability matrix 
interval transition probability matrix 

with respect to 

left (right) hand side 

asymptotic to 

convolution of two random variables X and Y 
convolution of two distributions with d.f. F, and F, 
convolution of n iid. random variables having 
common d.f. F 

indicates the end of a proof 


GLOSSARY OF COMMONLY 
USED NOTATIONS 


Chapter 1 

P(s) probability generating function (p.g.f.) of a г.у. 

Sy sum of Ni.id. random variables X;, where N isa 
random variable independent of X; 

G(s) p.g.f. of N 

Ек r.v. having Erlang-K distribution 

ts) = | е-ч dF(x) (ЫТ. of a r.v. with d.f. F) 

0 

tj eigenvalue of a matrix 

P(A) spectral radius of the matrix A 

x (x) column (row) vector 

x right eigen vector 

y left eigen vector 

P stochastic matrix 

Chapter 2 

{Х()}; {Xi} stochastic process in continuous time 

{Xn} stochastic process in discrete time 

Сы; CÒS, £) covariance function between X(t) and X(s) 

Chapter 3 

Pik transition probability from state j to state k 

Р = (рж) t.p.m. with elements руа 

Be m-step transition probability from state j to state k 

pim) t.p.m. with elements pt? 

fo probability that the system reaches the state k for 
the first time after n transitions, given that it started 
with state j 

Еһ (= 5 ff), і.е. probability that starting with state 


j, the system will ever reach state k 
thik expected first passage time from state j to state К 
P,,(8), Е(5) p.g.f. of (pf?) and { f (9) respectively 


368 


tj, Xp Yr 
vj 

V(s) 
Chapter 4 
N (t) 


Prt ) 
P(s, t) 


M(t) 


Qs. t) 
OQ» 


pit); рК) 
P(t) 


Уу 
aij 
A 


Chapter 5 
Qa» 

Ur 

e*t) 
M(T) 


T, 

U(t) 

m(t) 

c(t) 
Chapter 6 
Xn 
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eigenvalue, right and’ left eigenvectors respectively of P 
= lim pf? as n > © 


p.g.f. of (vj) 


number of occurrences of an event by time t (or 

upto epoch f) 

Pr {N(t) = n | NO) = -} 

p.g.f. of (N(t), t > 0} or of ( p(t} 
ми) 

compound Poisson process, ie. X X, where Xj, are 
i=l 

i.i.d. r.v’s and Mt) is a Poisson process 

p.g.f. of a non-homogeneous Poisson process 


Markov process in continuous time with discrete 
state space or continuous time Markov chain 


probability of transition from state ito state j by time t 
t.p.m. with elements Pit) 

lim pj; (t) as t — © 

transition density, i.e. Pu (£) lizo 

rate matrix or transition density matrix whose ele- 
ments аге (а) 


Wiener process 
mean-value function of X(t) 
variance function of X(t) 
maximum of X(t) in 0 < t < T( max X(t)) 
o<t<T 
first passage time to a fixed point а of a Wiener 
process with X(0) = 0, р = 0 
Ornstein-Uhlenbeck process (O-U.P) 
mean-value function of U(t) 
variance-function of U(t) 


non-negative independent r.v. (interarrival time 
between (n — 1)th and nth renewal) having mean p 
d.f. of X, 
nth renewal epoch, i.e. Sn = ix 

ici 


number of renewals in [0, t] (W(t) = sup (n: Sn < t 
expected number of renewals in [0, ¢] 
L.T. of M(t) 
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^ 


N4(t, T)(Ns(t, Т)) number of renewals in [0, г] under age (block) replace- 


G 
Np (t) (Ne (0) 


F(t) 


G(t, 2) 

Gp (t,.z) (С. (t, 2)) 
G* (s, z) 

M,(t) 


Y(t) 
Z(t) 


Yn 
{Xn, Yn} 
Y(t) 


са) 


Chapter 7 
{Х@)) 

Th 

Xn = X (tn +) 


Tn 


{Xm tn} 
Y, 


Qj (t); Qo 


Pik 
T; 


W(t) 
% jk 
al) 


ment policy with replacement interval T 

d.f. of initial distribution of a modified renewal 
process 

number of renewals in [0, t] for modified (equili- 
brium) renewal process 

equilibrium distribution of F having d.f. 


[а — од} ањ 


o 
p.g.f. of (N(t)) 
p-g.f. of Np (t) (Ne (t) 
І.Т. of G (t, 2 
renewal function of an alternating (or two-stage) 
renewal process (i — 0, 1) 
residual or spent lifetime ( forward recurrence time) 
at time (epoch) г, i.e. Y(t) = Snoey — t 
age (backward recurrence time) at time t, i.e. Z(t) = 
t — Sy(t) 
reward or cost associated with nth renewal 
renewal reward process 
total (cumulative) reward earned or cost incurred 
by time ¢ 
expected cumulative reward (or cost) by time t 


Markov process 

nth transition epoch 

state at transition occurring at epoch f; 

time interval (tj, — tm) between nth and (n + 1)st 
transitions 

Markov renewal process 

(= X, on f, € t < tma) i.e. state of the process 
at its most recent transition 

Pr {Xna = ky tr «1|X,—J) 

lim Qj, (t) as t — oo 

(conditional) waiting time in state given that the 
next transition is to state k 

(= Qjx(t)/pjx), i.e. d.f. of Тук 

expectation of Tj; 

rth moment of Тук 


24(u5-05/1982) 


шї 
fut) 


Qf? (t) 
м@, jt) 
F 


Jus fi 
fu*G) 
IU 


U(t) 


W(t) 


Chapter 8 
X 


FG) 
С(2) 
S (2) 
X(t) 
R) e) 
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unconditional waiting time in state j 

PríT; t) 

expectation of Т, 

rth moment cf 7; 

successive epochs t, for which X, — j 

number of transitions to state j (renewals of state j) 
in [0, t] 

= (N,(1), Nu), 2 

first passage time into state j from state i 

d.f. of Vi; 

mean first passage time into state j from state ї 
conditional probability that state at epoch ¢ is j 
given that initial state was i 

= рг (Х, = < 01 0 

Markov renewal function (= E(N;(t)| Xo = iJ) 
interval transition probability matrix (i.t.p.m) having 
elements fi; (t) 

limit fj) (t) as t — оо 

L.T. of fi; (t) 

stationary distribution of embedded Markov 
chain {Xn} 

(=t—t,) time since last transition (backward 
recurrence time) 


(= һы — tn) time to next transition (forward re- 
currence time) 


wide-sense or covariance stationary process in 
discrete time 


covariance function of X, 

correlation function of x 

purely random process 

backward shift operator (і.е. ВХ, = X.) 
spectral density function of X, 
autocovariance generating function of X, 
autocorrelation generating function of X, 
stationary process in continuous time 
covariance (correlation) function of X (t) 
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Chapter 9 


{рь} 
[4 
P(s) sun 


m, c! 

Xn 

Xo 

{Xn n > 0} 
Р, (з) 


by 

B, (s) 

B(s) 

*(0) 

G.W.I. Process 
h(s) 

a 

X (n 

P,(S) 

F(s) 


pr, Py. Tk) 

X, —(X,,, ...Xn'#)) 
т) 

X(t) 

pit) 


F(t, s) 
a; 


371 


offspring distribution 

i.i.d. random variable with distribution {px} 

p.g.f. of offspring distribution {px} and also of r.v. 
(1 n D MER 

mean and variance respectively of ( Pd 

size of nth generation, n — 1, Duros 

size of 0-th generation or ancestor 

Galton-Watson (G.W.) process 

p.g.f. of {Xn} 

probability of extinction 

total number of progeny upto nth generation in- 

cluding the ancestor, i.e. Y, = 1 + 2, 

p.g.f. of {Zn}, Za = X, 4-...4+ X, 

p.g.f. of (Y,) 

lim A, (5), as n — co 

Pr {X,=k|X, > 0) for a subscritical G.W. 

process 

lim bgn, as  — со 

p.g.f. of {bkn} 

p.g.f. of (b) 

lim Pr (X, > 0}/m", as n + со 

Galton-Watson process with immigration 

p.g.f. of immigrant distribution 

mean of immigrant distribution 

size of nth generation of a G.W.I. process 

p.g.f. of (Xu) 

lim P,(s), as n> оо 

probability that an individual of type i produces r, 

offspring of type j, j 2 1, 2,...k (in case of a multi- 

type G.W. process) 

population size of k types in the nth generation of 
a multitype G.W. process 

expected number of offspring of type j produced by 
an object of type i 

continuous time branching process 

Pr {X(t)=j| X(0)=i} fora Markov branching process 
p.g.f. of (pi; (1) 

transition density of {X(t), t> 0}, X(0) = 1 


F(t, s) 


Chapter 10 
^U 


Я 
N(t) 
Wa 
W(t) 


M 
G 
Pa(t)( Pin(t) ) 


Pn 
L(Lo) 
W(Wo) 
ws) 

fals) 
P(z, t) 
T 

N* (1) 


МӘ | 
Ma 

Pas Porn) 
Pysn (Ро) 


M/G/1 
B(t) (b(t) 
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p.g.f. of (aj) 

lim ру (f) as t— co, i.e. (probability of extinction) 
d.f. of lifetime of objects in an age-dependent 
branching process ке 

p.g.f. of an age dependent branching process (X(0) 
with X(0)—1 


mean arriva! and service rates respectively 

traffic intensity 

number in the system at epoch t 

waiting time of nth arrival 

virtual waiting time, i.e. duration of time a unit 
would have to wait were it to arrive at epoch 
(instant) t 

exponential (Markovian) distribution 
arbitrary (general) distribution 
probability that the number in the system at / іѕл 
given that number at t:=0 is zero (is i) 

lim p,(t) as t— oo 
expected number in the system (queue) 

expected waiting time in the system (queue) 

p.d.f. of waiting time in the system (queue) 

L.T. of pa(t) 

p.g.f. of (p,(0] 

busy period initiated by a single customer or unit 
number in the system at epoch : during a busy 
period 

arrival streams follow exponential distribution, 
number arriving at an arrival instant being given 
by a r.v. X 

bulk service with exponential service time, under a 
general bulk service rule (i.e, a batch taking a 
minimum of а units and a maximum of b units) 
probability that the number in the queue at time ¢ is 
n, the service channel being busy (idle) under general 
bulk service rule 

limiting value of pi, (£( pos(£)) as (9% 


d.f. (p.d.f.) of general or arbitrary service time v 
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B*(s) 


tn 
Xn=N(ty +0) 


kr 


Pij 
ИЛ 
V(s) 
K(s) 
W(Wao(t)) 


W*(s) 


бумл 
A(t) (a) 


A*(s) 


fn 
Y, N(t,—0) 


Z(t) 


Pij 
Er 


G(s) 
{vj} 
(2) 
To 
Nt) 
fi) 
Pii(t) 


Reliability 
Xi 
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| e^* dB(t) 

0 

nth departure epoch 

number in the system immediately after mth 
departure 

probability that r units arrive during service time 
of a unit 

Pr {лы = Xi) 

stationary distribution of (X,) 

p.g.f. of (vj 

p.g.f. of {К} 

d.f. of waiting time in the system (queue) 


со 


[ e*t айп) 


0 


d.f. (p.d.f.) of general or arbitrary independent 
interarrival time u 


Í e! dA(t) 

0 

nth arrival epoch 

number in the system immediately before the arrival 
of the nth unit 

(=У,, t, St < 11+) system size at the most recent 
arrival 

=Pr {Ynn =j | Yn=i} 

probability that number served in the inter-arrival 
time between nth and (n--1)st units is r 

p-g.f. of . gr} 

stationary distribution of (Y;) 

=:—А%(—ш) 

unique root of r(z)=0 lying inside | z j21 

system size at an arbitrary epoch ¢ 

Pr{Z(t)=j| Z(0)=i} 

Pr(N()-jI NO)=i} 


Berrioulli r.v. with p.f. Pr(Xi—1) = pi, 
Pr (X,—-0)—1—pi 
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=min (X,,.-., Xn) 

lifetime of a device or equipment 

reliability of the device i at time / 

(—Pr (Tj > 1) 

hazard rate: (— — R;'(t)/ R(t) ) 

reliability of a system consisting of п independent 
components 1, 2,..., п 

mean of exponential distribution 

accumulated damage by time t 
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TABLE OF LAPLACE TRANSFORMS 


Laplace transform of f(t) is defined by 


оо 


{До} = [eo dt 


0 


A) LIRO 
1 
1. 1 ; 
2 11-1 aN se РГ D. 
Ss 
3 p KOI А 
5 
4 i dm 
` vt) MS 
5 eat 1 
2 Bic 
1"—1 eat 1 Е 
ms (n — 1)! Era S 1,2, . 
a 
7. sin at ET 
5 
8. cos at "EET 
9, f(t)=0,0<t<a scs 
elati 5 
(Heaviside unit function) 
10. d(¢ — a) enas 
(Dirac8 function located at a) 
A 
Ш: Лем (А > 0) Ti 
k 
X te eM 0 k [ A 
bk >0 ——- 
12; ro ©” ) | 
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sep) exp (— a4/(2s)) 


Is — (5° — a)" 


14. а" I, (at) /(s? — а) ° п> —1 


15. т Ip (at) [s —V/(s? — a)", п = 1, 2, ... 


16. {V(t} In (2/1) oe nus 12; «+ 


Sa. 
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Subject Index 


Abel’s lemma, 66 

Addition of random variables, 5, 1,22, 
23 

Additive process, 160 

Age-dependent branching process, 

267-271, 274 

definition of —, 267 
expectation of —, 269 
extinction probability of —, 271 
generating function of —, 268 
multi-type —, 271 

Ancestor, 241, 318 

Arrival epoch system size, 324 

Arrival rate, 278 

Autocorrelation function, 231 (see corre- 
lation function) 
—generating function, 231-232 

Autocovariance function, 106, 231 (see 
covariance function) 
—generating function, 231-232 

Autoregressive integrated moving average 
process (ARIMA), 236 

Autoregressive moving average process 
(ARMA), 229 

Autoregressive process, 226 
—of order one, 233 
—of order two, 228 

Availibility, 342, 349 


Basic limit theorem of renewal theory, 
66 
Bellman-Harris process (see age-depen- 
dent branching process) 
Bernoulli process, 45 
Bessel function, 40 
—density, 40 
Binary splitting process. 273 
Biology, 122, 272 
Biorthogonal system, 34 
Birth and death process, 120-122, 143, 
265, 294-295 
Birth and death rates, 122 
Bochner-Khinchine theorem, 231, 235 
Box-Jenkins method, 236 
Branching process, 92, 241-276, 319 
age-dependent—(Bellman-Harris 
process), 267-271 


D 
continuous time Markov —, 262-267 
continuous time multi-type —, 271 
extension and generalisation of conti- 

nuous time —, 271-272 

Galton-Watson, —, 241-262 
general —, 271 
Markov renewal —, 330 
multi-type age-dependent —, 271 
multi-type Bellman-Harris —, 271 
Sevast'yanov type — 271 
—with disasters, 272 

Brownian motion, 146 
—process, 146, 148 

Busy cycle, 197 

Busy period, 197 290, 317 
— and branching process, 318 
number served in a —, 320 
— of M/G/1 model, 317, 320 
— of M/M/1 model, 290-293, 320 
— of M/M*+#/1 model, 346 


Central limit theorem, 146 
Chapman-Kolmogorov equations, 60-62, 
85,101, 127-129, 147, 262, 264, 314, 
336 
— for Markov chain (with continuous 
state space), 85 
— for Markov chain (with discrete 
state space), 60-62 
— for Markov process (with continuous 
state space), 147 
— for Markov process (with discrete 
state space), 127-129 
Characteristic function, 2 
Classical ruin problem, 27 
Classification of Markov chains, 63 
aperiodic, 70, 74, 75, 77, 92 
denumerable, 60, 72, 76 
ergodic, 65, 74, 75 
homogeneous,’ 61, 79, 85, 90 
irreducible, 64, 68, 69, 74-78, 92 
periodic, 72 
reducible, 64 
Classification of states of a Markov 
ehain, 63 
absorbing states, 6% 
apériodic states, 65, 67, 68 
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ergodic, 65 

non-null persistent state, 65-69, 74-77 
null persistent states, 65-70 

periodic states, 65, 67, 68, 69 
transient states, 65-69, 77 


Cobb-Douglas production function, 302 
Collective risk theory, 114 є 
Communication system, 62, 90 
Conditional expectation, 9 
Conditional limit laws of G.W. branching 
process, 253-258, 272 
Conservation method, 280 
Conservation principle, 280 
Consumer brand switching model, 93 
Continuity theorem of Laplace trans- 
forms, 21 
Continuous time Markov branching 
process, 262-267, 269 
definition of —, 263 
generating function of —, 263 
probability of extinction of —, 266 
Continuous time Markov chains, 126-137 
Convolution, 6, 21 
— theorem of Laplace transforms, 21 
Correlogram, 224 
Correlation function, 106, 224, 227 
properties of functions, 230 
-Counting process, 97 
Covariance function, 48, 50, 224, 230, 
234 
properties of functions, 230 
Covariance stationary (process), 48, 50, 
51, 63, 224, 226, 228, 230 
Cyclic component, 223 


Demography, 272 

Departure epoch system size, 313, 329 

Difference equations, 24-30, 41 
characteristic equation of —, 26, 27 
differential —, 30-33, 42 
homogeneous —, 25 
— in probability theory, 27-30 
non-homogeneous —, 29 

Diffusion coefficient, 151 

Diffusion equations, 147, 149 
forward —, 149 
backward —, 149 

Diffusion processes, 146 

Dirac function, 17 

Direct descendants, 241, 318 

Directed graph, 77 
arc-weight of an intree, 137 
intree to a point in a —, 78 
weight of an intree, 78 
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Distribution, 
Bernoulli —. 39, 331 
binomial —, 4 
chi-square —, 20, 81-83 
compound Poisson —, 9, 39 
decapitated (modified) geometric —, 38 
Erlang —, 20, 22, 138-139 
exponential —, 4, 17, 18, 22, 24, 40 
gamma —, 19, 22 
inverse Gaussian —, 155, 156 
geometric —, 3, 7, 38, 39, 113, 186 
geometric and exponential —, 111 
lattice —, 176 
mixed exponential, 167, 178, 200 
modified exponential —, 41 
multivariate normal —, 49 
negative binomial —, 7, 39, 113, 162, 186 
negative exponential —, see exponen- 

tial— 
non-lattice —, 176 
normal —, 49 
Pascal —, 7 
Poisson —, 3, 6, 17, 39, 40 
special kind of discrete —, 17 
triangular —, 41 
uniform (rectangular) —, 18, 41 
zero-truncated (decapitated) Poisson, 
39, 40, 116 
Distribution of total number of progeny, 


252-253, 273 
Dynamic indeterminism, 45 


Ecology, 122 
Economics, 122, 146 
Eigenvalue, 33 
Eigenvector, 33 

left —, 34 

tight —, 34 
Entrance probabilities, 220 
Epidemiology, 272 
Erlangs, 278 
Erlang's first formula, 299 
Erlang's loss formula —, 135, 137, 299 
Erlang process, 138 
Euler's constant, 335 
Extinction of families, 241 


Fatou's lemma, 74 

Fibonacci numbers, 27, 38 

First passage (time) distribution, 65, 152, 
209, 293 

Flow graph formula, 77 

Fokker-Planck equation, 151 

Fourier-transform, 2 

Frequency domain, 230 
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Galton-Watson branching process, 241- 
262 
conditioned limit laws ӨЁ —. 253-258, 
272 
critical —, 244, 253 
definition of —, 241 
generalisation of the classical —, 258- 
260 
generating function of —, 242-244 
— in varying and random environ- 
ments, 260-261 
moments of —, 243-244 
multitype —, 261-262 
probability of extinction of —, 244- 
248 
subcritical —, 244, 256 
supercritical —, 244 
time to extinction in a —, 274 
total number of progeny in —, 252- 
253 
— with immigration, 258-260 
Gamma process, 161, 162 
Gaussian process, 49 
General (random) epoch system state, 
326 
General branching process, 271 
Generating functions, 1-12, 29 
bivariate probability —, 10, 40 
factorial moment —, 2 
moment —, 2 
— of bivariate distribution, 9 
— of branching processes, 242-244, 
262 
— of renewal processes, 181-182 
— of sums of random variables, 5-9 
probability —, 1-12,.42- 
Graph theoretic approach, 77, 136, 144 
Graph theoretic formula for finding 
stationary distribution 
— for Markov chain, 77-79 
т continuous time Markov chain, 
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136-137 


Hazard rate, 333 
— average, 333 
increasing — (IHR), 333, 341 
increasing — average (IHRA), 333, 
341 
Heat equation, 150 
Heavy traffic, 330, 331 


Idle period, 197, 292 
Immigration-death process, 126 
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Immigration-emigration process, 123 
Input, 277 
Integrated spectrum, 231 
Interarrival time, 107, 108, 164, 277 

— in a renewal process, 164 
Intermittently available server, 305 
Interval transition probability matrix, 211 
Inverse Gaussian process, 161 
Inventory model, 92 


Jirina process, 272 
Key-Renewal theorem, 176, 210, 270 


Ladder processes, 331 

Laplace transform, 12-24, 41, 42 
continuity theorem of transforms, 21 
convolution theorem of transforms, 21 
inverse —, 14 
— of Dirac function, 17 
— of probability distributions, 15-24 
properties of transforms, 13-14 
uniqueness theorem of transforms, 21 


Laplace-Stieltje’s transform, 25 


Lattice random variable, 176 

Lifetime distributions, 333, 334 

Likelihood ratio criteria, 81 

Linear growth process, 123-125, 141, 142 
extinction probability in —, 125 
—with immigration, 126, 142 

Little’s formula, 280, 284 

Loss and delay system, 344 
combination of —, 344 

Loss function, 83 

Loss system, 134, 277, 299 

Lotka’s Model, 250 


MAICE, 82-83 
Machine interference problem, 135, 302, 
345 
Maitainability, 342 
Malthusian parameter, 271 
Markov branching process, see conti- 
nuous time Markov branching process 
Markov chain, 48, 54-95 
Chapman-Kolmogorov equation of —, 
60-61, 62, 85 
classification of chains, 63-64 
classification of states of chains, 63 
continuous time chains, 126-137 
definition of —, 56 
denumerable (or denumerably infinite) 
—, 60, 72 
determination of the order of —, (by 
MAICE), 82-83 
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general existence theorem of chains, 58 
graph theoretic approach for finding 
stationary distribution of a —, 71-18 
grouping of states of a —, 92 
higher transition probabilities of a —. 
60, 69 
homogeneous —, 56, 61 1 
homogenous (stationary) transition 
probabilities of a —, 56 
hypothesis testing in —, 81-82 
irreducible —, 64, 68, 69, 74-78, 92 
limiting behaviour of —, 70, 74 
maximum likelihood estimation of 
transition probabilities of a—, 79- 
82 
nonstationary (nonhomogenous) 
chains—, 56, 87-88, 90 
— of order one, 58, 82 
— of order zero, 8 
order of a —, 58, 72, 81 
state space of chains, 56 
Stationary (invariant) distribution of 
a —, 73, 74, 75 
statistical inference for chains, 79-84 
three state —. 71, 72, 75 
transition probability matrix of a —, 
54 
two state —, 71, 75 
— with continuous state space, 84-87 
Markov process, 47, 51, 126, 146, 214, 


215 
definition of , 48 


ergodicity of homogeneous —, 133 
three-state —, 144 

two-state —, 131, 134, 349 

— with continuous state space; 146- 


163 
— with discrete state space, 96, 126- 


137 
Markov renewal equation, 210-211 
Markov renewal function, 209-210 
Markov renewal process, 195, 203-222, 
326, 342 
definition of —, 204, 208 
first passage time in —, 209, 216-218, 
220 
interval t.p.m. of —, 211 
recurrent —, 210 
states of —, 210 
theory of birth as a —, 204 
transient —, 210 
Markov system, 73 
Markov tinié (stopping time), 170 
Matrix, 33 
cau$ative, 87-90 
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characteristic equation of a —, 33 
characteristic (latent) roots of —, 33 
constituent matrices, 36 
diagonal —, 36, 70 
doubly stochastic —, 37, 43, 92 
eigenvalues (spectral values) ofa —, 
33, 34, 38, 43 
eigenvectors of a —, 33, 34, 43, 70 
imprimitive (cyclic) —, 35 
irreducible —, 35, 38, 64 
non-negative matrices, 35 
permutation matrices, 35, 43 
primitive matrices, 35, 38, 43, 70-72 
reducible —, 35 
spectral radius of a —, 33 
spectral representation ot a —, 36, 72 
spectral theorem of —, 70, 71, 129 
spectrum of a —, 33 
stochastic (Markov) —, 37, 38, 57 
trace of a —, 34 
Meteorology, 223 
Models of time series, 224 
autoregressive process, 226, 228 
autoregressive moving average process 
(ARMA), 229 
continuous parameter process, 234 
first order Markov process, 233 
integrated autoregressive and moving 
average process (ARIMA), 236 
moving average (MA) process, 225 
non-stationary process, 236 
white noise process, 224 


Negative binomial process, 162 

New Better than used (NBU), 181, 200 
— in expectation (NBUE), 181 

Newton-Raphson method, 325 

New worse than used (NWU), 181 


— in expectation, (NWUE), 181 
Noise, 223 


Non-Markovian chain, 62, 92 
Nuclear physics and Chemistry, 272 
Number of renewals in a random time, 


185, 200 
average —, 170 


Occupancy problem, 91 
Offspring distribution, 241 
One-minus-one process, 62 
Operational time, 119 

— interval, 119 
Operator 

backward shift —, 225 

displacement —, 25 
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156-158, 
161 
— asa transformation of Wiener 
process, 159 
Output process, 343 


Ornstein-Uhlenbeck process, 


Perron-Frobenius theorem, 35 
Poisson cluster process, 113-114, 141 
Poisson counting process, 112, 186 
Poisson increment process, 52 
Poisson process, 49, 96-120, 130, 138, 
139 
— and related distributions, 107-110 
autocorrelation function of —, 106 
characterisation of —, 107 
compound —, 106, 113-115, 341, 342 
decomposition of a —, 106 
deterministic sclection of —, 139 
doubly stochastic —, 120 
estimation of the parameter of a —, 
101 
generalisation of —, 113-120 
— in higher dimensions, 113 
interarrival time of a —, 107, 108 
linear combination of independent 
processes, 114 
mixed —, 162 
non-homogencous (time-dependent) —, 
101, 118, 119, 342 
postulates for —, 97-98 
properties of processes —, 102-106, 
110-112 
random selection of —, 139 
random variation of the parameter of 
—, 119 
waiting time or —, 108 
Pollaczek-Khinchine formula, 313, 314- 
315 
—mean value formula, 317 
— transform formula, 315, 317 
Polya process, 119 
Polya's urn model, 54, 55 
Power spectrum, 230 
Probability of (ultimate) extinction or 
extinction probability, 
— in a continuous time branching 
process, 266-267 
—ina G.W. branching process, 244- 
248 
— in a linear growth process, 125, 142 
— in an age-dependent branching pro- 
cess, 271 ` 
Probability vector, 43 
Progeny, 252, 318 
Programming procedure, 79 
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Pure birth process 116, 141 
Pure death process, 126, 141 
Purely random event, 109 


Queue discipline, 277 
Queueing models, 
D|M]1, 326 
Ex M1, 325 
GI|M]A, 323-325 
MID/1, 314 
M]Er|1, 313 
M/G/1, 208, 311-313 
MIG%4/1, 347 
M/G%>/1, 327-328 
МІС[оо, 321-322 
МІСІ]5, 345 
M(X)/G226)1, 347 
МүМ]\, 206, 215, 280-293 
M/M|s, 295-298 
ММ] оо, 279, 289-299, 320-322 
ммК, 284-286 
M[G/s!s, 300 
M/M]s/s, 299-300 
MIM?]1, 329 
M/M2>5/1, 305-310, 329 
міма», 347 
M/M*4/1, 347 
Queueing processes, 86, 197, 278 
non-birth and death —, 303 
Queues, 
average cost in batch service —, 348 
bulk —, 303 К 
bulk service —, 304-310 
centralisation of counters in —, 343 
combination of loss and delay system 
in —, 344 
group arrival —, 303 
martingale approach in —, 331 
multichannel —, 294-300 
multistage Poisson queueing system, 
346 
optimal design and control in —, 331 
Wiener-Hopf techniques in —, 330 
— with finite waiting space, 284-286 
— with limited input source, 301-302 
— with limited waiting time, 300 
zero-avoiding transitions in —, 343 
Random function (or process) see 
stochastic process 
Random geometric sum of exponential 
variables, 24 
Random modification (or residual time), 
110 


Random selection, 139 
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Random transmission 
(semi-binary), 52 
Random variables, 1, 5, 7, 22, 23, 176 
integer-valued —, 1 
lattice —, 176 
sums of —, 5, 7, 22, 23 
Random variation of the parameter of a 
Poisson process, 119 
Random walk, 59, 147 
continuous limit of simple —, 147 
— having continuous r.v. as steps, 85 
— with absorbing barriers, 59 
— with reflecting barriers, 76 
Recurrence time, 65, 187-189 
backward —, 187-188 
forward —, 187-189 
Reflection principle, 152 
Regeneration point, 164, 311, 326 
system state at —, 326 
Regencrative stochastic process, 195, 196 
existence of limits of —, 195 
Reliability, 332-342 
application of M/M/oo queueing 
model in —, 322 
k-out-of п system —, 332 
Markovian model in —, 336 
— optimisation, 342 
parallel system, 332, 338 
parallel system with maintenance, 338 
series system —, 332 
shock models in —, 339-342 
system —-, 334, 335 
system reliability in random environ- 
ment, 349 
— theory, 122, 203 
wear processes in—, 339-342 
Kcnewal argument, 168 
Renewal (or regeneration) epoch, 164 
Renewal equation, 167-169 
solution of —, 169 
Renewal function, 165 


L.T. of —, 165 

— of age and block replacement 
policies, 174 

— of equilibrium renewal process, 
179-180 


Renewal process, 120, 164-202 
alternating (or two-stage) —, 192-195 
definition of —, 164 
delayed (modified or general) —, 177- 

179, 189, 193 
equilibrium (stationary) —, 179 
interarrival time in a —, 164, 166 
ordinary —, 177, 178, 216 
p &.f. of processes —, 181 


SUBJECT INDEX 


renewal rate, 179 
uniform —, 189 
Renewal reward process, 190-192, 348 
age-based —, 191, 201 
definition of, 190 
Renewal theorems, 169-177, 216 
Blackwell’s —, 176, 210 
elementary -—, 170, 172 
key (Smith’s) —, 176, 20 
Renewal theory, 164 \ 
application to a management problem, 
183 
basic limit theorem of —, 66 
generalisation of the classical —, 
198-199 
integral equation of —, 168 
— in queueing process, 197 
Renewal type equation 169, 188 
Replacement interval, 174 
Replacement policies, 173-175 
age —, 173 
age-based renewal processes under—, 
191 
block —, 174, 202 
number of removals under —, 200 
renewal functions under —, 174 
renewal processes under —, 174, 175 
Residual and excess (or spent) life times, 
186-187, 190, 216 
Rouche's theorem, 290, 306 


Sample paths (or trajectories), 
— of a stochastic process, 152 
— of a Wiener process, 152 
Schroder functional equation, 258 
Seasonal components, 224 
Semi-Markov process, 195, 203-221 
definition of —, 204 
embedded Markov chain of —, 204 
— in queues, 206-208, 215, 221, 313 
limiting distribution of —, 212, 
215-216 
recurrence times of —, 215-216, 220 
waiting time in —, 205 
Service time, 278 
Shock models, 339-342 
Signal, 223 
Spares inventory model, 322 
Spectral density function, 231 
estimation of —, 237 
normalised —, 231 
Stability of a Markov system, 73 
State space, 46 
discrete —, 46, 47, 56, 84, 85 
\ continuous —, 46, 47, 84-85 


SUBJECT INDEX 


Stationary process, 48 
covariance, —, 48 
Steady state distribution, 279 
Stochastic difference equation, 225 
Stochastic processes, 45 
covariance function of —, 48-51 
covariance stationary (weakly or wide 
sense stationary), 48-53 
continuous parameter, 47 
discrete parameter, 47 
evolutionary, 49, 50 
Gaussian, 49 
multi-dimensional, 46, 47 
one-dimensional, 46, 47 
one-minus-one, 52 
specification of, 46 
state space of, 46 
strictly stationary, 48, 49, 51 
— with independent increments, 47 
Stochastic sequence, 47 
Stopping time, 170, 171, 172 
Supplementary variable technique, 330 
System state 
arrival point —, 324 
departure point —, 311, 326 
general time —, 313, 326, 329 


Tackacs integral equation, 317-318 
Tauberian theorems, 14 
Telegraph signals 
random —, 52 
semi-random —, 52 
Telephone traffic, 277 
Threshold, 341, 342 
Time domain, 230 
Time series, 223-240 
continuous parameter —, 234 
models of —, 224-230 
statistical analysis of —, 235 
statistical inference in —, 235 
Traffic intensity, 278 
Transient behaviour, 
— of M/M/1 queue, 286-293 
— of M/M/1/1 queue, 302-303 
— of M/G/oo queue, 320-322 
Transition density, 127, 128, 137 
— of continuous time branching 
process, 263-264 
— of continuous time Markov chain, 
+127, 128 
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— (rate) matrix, 128, 135, 144 
Transition probability, 56 
higher—, 60 
— matrix, 57 
Trend, 223 
Tshebyshev's lemma, 102 


Uniqueness theorem of Laplace trans- 
forms, 21 


Virtual waiting time, 293 


Waiting time, 108, 205, 282 
— for a change of state, 127 
— for the first success, 4 
— process, 293 
virtual —, 293 
Waiting time distributions, 282 
— in GI/M]1 queue, 325 
— in M/M/I queue, 282.284 
= in M[M^?:5]1 queue, 307-310 
— in M/Mjs queue, 296-297 
Wald's equation, 170-171 
Weakly (wide-sense) stationary, 48 
Wear proeess, 339-342 
White noise process, 224 
Wiener-Hopf technique, 330 
Wiener-Einstein procees, 147 (see Wiener 
process), 147-156, 159 
backward diffusion equation of —, 149 
definition of, 148 
differential equations for a —, 149 
distribution of maximum of —, 142 
drift of —, 148 
first passage time distribution of —, 
152-156 
forward diffusion equation of —, 149 
Kolmogorov equation of —, 150 
sample path of —, 152 


Yaglom's theorem, 256, 259, 260 
Yule process, 228 

Yule-Furry process, 116-117, 142, 143 
Yule-Walker equations , 228 


Zero-avoiding state probability, 290 
Zero-truncated Poisson distribution, 39,. 

40, 116 
Zipf's rank size law, 141 
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